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ABSTRACT 

This report presents results from an internal evaluation 
conducted between September 1981 and August 1982 of abstracts published by 
the Educational Resources Information Center (ERIC) system. A brief 
introduction describes the study methodology and reviews limitations, 
indicating that 526 evaluations of 349 abstracts were conducted by personnel 
at each of the 16 ERIC Clearinghouses. Next, results of the evaluations 
prepared by each Clearinghouse are presented, with Clearinghouses being 
identified only by a random letter of the alphabet. For each Clearinghouse, 
information is provided on the numbers of abstracts evaluated, evaluations 
completed, abstractors whose work was represented, and evaluators; overall 
ratings; types of abstracts represented (i.e., informative, indicative, or 
mixed) and analysis of their appropriateness; evaluations of abstract style, 
vocabulary, and content; length of the abstracts evaluated; and details on 
how the Clearinghouse conducted the evaluations. Next, outcomes are described 
for the system as a whole, indicating that 67 excellent ratings, 188 good 
ratings, 150 fair ratings, 91 marginal ratings, and 30 poor ratings were 
assigned. Common problems included inappropriate selection of abstract type 
and the omission of important sections of documents. Recommendations for 
improving abstracts are included. An interim report from an initial pilot 
study; revisions of the abstract evaluation form; feedback from 
Clearinghouses regarding the form; sample evaluations; and materials from an 
abstracting workshop are appended. (BCY) 
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BEST COPY AVAILABLE 



The Abstract Evaluation Project was initiated in spring 1931, when Central 
ERIC contacted the ER i C C 1 ear inghouse for Junior Colleges and requested that 
the Cl ear i nghouse design and implement a project to evaluate abstracts currently 
being written for the database. During summer 1981, the project was designed, 
and in September the first evaluation instrument was pilot tested. Between 
September 1 98 1 and August 1982, during two pilot testings of evaluation forms 
and three cycles of evaluations, a total of 3^9 abstracts were evaluated and 
526 evaluations were completed. 

The following sections provide syntheses of these evaluations by Clearinghouse 
and for the system as a whole. While these composite evaluations provide a 
fair idea of abstracting in the ERIC system, certain limitations should be kept 
in mind wh i 1 e reviewing these findings: 

1) While the evaluation instruments requested the evaluators to 
look carefully at various aspects of the abstracts, they 
provided no set criteria for assigning "excellent," "good, 11 
"fair," etc. ratings. Therefore these ratings were assigned 
on the basis of personal criteria and are far from standard. 

2) Certain Clearinghouses were very lenient in evaluating abstracts, 
while others were extremely demanding. Certain evaluators focused 
almost exclusively on writing style, while others tended to ignore 
style and concern themselves with content. 

3) Certain evaluators applied standards that had been developed at 
their Clearinghouse rather than applying systemwide guidelines. 

b ) The sample, which was drawn from five different months of ■ Resources 
in Education, is fairly small given the total number of abstracts 
that were processed during this period (i.e., 3^9 out of 6,259 abstracts). 



i 



A 



Number of A abstracts evaluated: 15 
Number of evaluations completed of A abstracts: 30 
Number of abstractors whose work is represented: 4 

Number of evaluations completed by A staff: 32 
Number of A evaluators: 1 



Qveral 1 Rat \ ngs : 

2 excellent; 7 good; 1 4 fair; 5 marginal; 2 poor 

Of the 30 evaluations completed, 9 provided ratings of good or excel len" 

Selection on Abstract Type ' 

Of the 15 abstracts evaluated, 2 used the informative mode, 10 used the indica- 
tive mode and 3 used the mixed mode. In 8 cases the evaluators found the choice 
of abstract type appropriate. In the 7 cases in which the evaluators didn't find 
the abstract type appropriate, they felt the informative or mixed mode would have 
been more effective. 



Style and Vocabulary 



Almost all of UD 1 s abstracts were rated quite highly in all areas of style 
and vocabulary. Only one of the 16 abstracts evaluated was considered 
confusing. The other abstracts were felt to be intelligible, logically 
structured, stylistically well written, and understandable to the non-specialist. 

Content 



While 9 of the abstracts evaluated covered the important topics of the 
document, the abstracts that received lower ratings shared what were 
considered major content problems. The most common criticism of the 
abstracts was that they provided superficial coverage (six cases). The 
evaluators felt that eight of the abstracts should have provided additional 
detail. In eight abstracts, major omissions were noted, such as important 
topics, research methodology and findings. One abstract was felt to mis- 
represent the document and in two cases inaccuracies were identified. 

Length: 



0-50 words: 0 
51 - 1 00 words : 4 
101-150 words: 4 
1 5 1 -200 words : 6 
200+ words: 1 



As most of the abstracts used less than 175 'words, the evaluators were especially 
critical of what they considered to be a superficial treatment of the documents. 




Clearinghouse a r . Evaluator 



All 32 evaluations were completed by the Clearinghouse's editor. She assigned 
2 excellent ratings, 17 good ratings, 10 fair ratings, 3 marginal ratings, and 
no poor rat I ngs . 

The consistently rather high ratings are somewhat surprising given the comments 
that accompanied some of them. In many cases abstracts were rated as good even 
when they were misleading about the document's content, had major omissions, 
or misrepresented the document. On the other hand, most of the fair or marginal 
ratings were given to abstracts considered to be poorly written, to over use 
professional jargon, even when the content areas were all adequately or completely 
covered . 

Of the 32 evaluations completed 8 m i s i dent i f i cd abstract type. 



B 



Number of B abstracts evaluated: 20 
Number of evaluations completed of B abstracts: 31 
Number of abstractors whose work is represented: 5 

Number of evaluations completed by B staff: 45 
Number of B evaluators: b 



O vera 1 1 Rat i ng s : 

5 excellent; 8 good; 9 fair; 8 marginal; and 1 poor 

Of the 31 evaluations completed, 13 a warded excellent or good ratings to the 
abst ract s * 

Selection of Abstract Typ e 

Of the 20 abstracts in the sample, G used an indicative, mode, 2 used an informative 
mode, and 12 used a mixed mode. Choice of abstract type was considered appropriate 
in 16 cases, and inappropriate in b cases. 

S tyle and Vocabulary 

For the most part the abstracts were rated highly in the areas of style and 
vocabulary. Several evaluators commented upon how well written the abstracts 
were. In only two cases did the evaluators comment that the abstracts were 
difficult to read or confusing. In only one Instance did the evaluators note 
terms that they felt should have been explained. Twice the evaluators commented 
that jargon had been overused, and in two instances gramma t i ca 1 problems were 
ident i f i ed . 

Content : 

Wh i 1 e in at least 13 cases the abstracts were felt to have covered all of the 
important topics in the documents, several problems with content were noted in 
the lower-rated abstracts. In seven abstracts, the evaluators pointed to what 
they considered major omissions, such as important sections or topics of the 
document, background information, examples that would have clarified the content 
of the document, and research methodology. The evaluators felt that five abstracts 
could have provided additional detail. In three cases, the evaluators indicated 
that the abstracts failed to give a good overall picture of the content of the 
document. In no cases were inaccuracies noted. 

Length : 

050 words: 0 

51-100 words : 2 

101-150 words: 7 

151-200 words: 9 

200+ words: 2 

Clearinghouse as Evaluator 

Four Clearinghouse staff members participated in the project, including two 
Abstractors, the Clearinghouse's Associate Director, who also writes abstracts, 
and the User Services Representative. They awarded 4 excellent ratings, 24 good 
ratings, T6fair ratings, and 1 poor rating. , 




Some of the ratings seemed rather high in light of the comments which accompanied 
several good and fair ratings, such as comments concerning major omissions, 
unintel 1 jgibi 1 i ty, and failure to provide a good picture of the document. 

The evaluators stressed the importance of comprehensive coverage and clearly 
stating the purpose of a study or document. They seemed quite concerned with 
abstracts over 200 words in length, especially when they felt that the abstract's 
length could be reduced by careful editing. 

Evaluators, especially the user services represents t I ve and the abstractors, 
seemed to have difficulty in determining abstract type. In the 36 abstracts 
they evaluated, they m i s i den t i f i ed abstract type 11 times. 
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c 



Number of different C abstracts evaluated; 19 
Total number of evaluations of C abstracts completed; 33 
Number of abstractors whose work is represented: 2 

Number of eva 1 ua * ‘ ans completed by C staff* ^+3 
Number of C evaluators: 5 



Qvcra 11 Rating s : 

5 excellent; 8 good; 13 fair; marginal; and 3 poor 

Of the 33 evaluations completed, 13 providing ratings of good or excellent. 
Selection of Abstract Type : 

Of the 19 abstracts evaluated, 6 used the informative mode, 1 used the 
indicative mode, and 12 used the mixed mode. In 1^» cases, the choice of 
abstract type was considered appropriate for the document. In five cases, 
evaluators seemed to feel that either the informative or indicative mode 
should have been used in place of the mixed mode. 

Style and Vocabulary : 

For the most part the abstracts were rated quite highly in areas of style 
and vocabulary. The abstracts were generally considered to be intelligible, 
to use correct grammar and spelling, to be understandable to the non-specialist, 
and to provide good topic sentences. In only two instances did evaluators feel 
that terms used in the abstract should have been explained. Only one abstract 
was felt to use overly long and complex sentences, anc! in only one instance 
were grammatical problems noted* 

Content : 

Several abstracts were felt to have covered all of the important topics in 
the documents, however, several problems were common to the lower- rated 
abstracts. In six abstracts, the evaluators pointed to major omissions, 
such as important sections of the document, research objectives or findings, 
and survey response rate in a case where the very low response rate affected 
the validity of the study. In three cases the evaluators felt the abstracts 
should have provided additional detail. Twice the evaluators felt the abstracts 
were not weighted appropriately In terms of the documents 1 emphases. In only 
one case did an evaluator feel that the abstract had failed to give a good 
overall picture of the document, inaccuracies were pointed out in two abstracts. 

Length : 



0-50 words: 0 

51-100 words : 1 

1 0 1 - 1 5 0 wo rd s : 7 

1 5 1 -200 words : 9 

200+ words: 2 



Evaluators tended to feel that the shorter abstracts (less than 150 words) should 
have provided additional useful information. 



C 1 on r inghousG as Evaluator 



Five Clearinghouse staff members participated in the project, the assoc-iate 
director, senior information analyst., two abstractors, and the user services 
coordinator. They assigned 6 excellent ratings, 12 good ratings, 1*1 fair 
ratings, 7 marginal ratings, and b poor ratings. 

The evaluators stressed the importance of comprehensive coverage of the 
documents' major topics. They tended to evaluate abstracts in terms of their 
usefulness, underscoring the importance of providing a clear idea of a document's 
contents and purpose. They were critical of abstracts which, through complicated 
sentence structure or poor writing style, are difficult to read. 

Of 39 evaluations, 12 mi s i den t i f i cd abstract type. 



D 



Number of D abstracts evaluated: 17 
Number of evaluations of D abstracts completed: 32 
•Number of abstractors whose work was represented: 5 

Number of evaluations completed by D staff: 38 
Number of D evaluators: 2 



Qvera 1 1 ratings : 

1 excellent; 18 good; 11 fair; 2 marginal; no poor 

Of the 32 evaluations completed, 19 provided ratings of excellent or good. Only 
two evaluations reflected inadequate abstracts. 

Style and Vocabulary 

The style and vocabulary of D 's abstracts were consistently rated highly. 

In only two cases were the abstracts considered unclear or difficult to 
read. Only one abstract was felt to contain unexplained terms. Many abstracts 
using an indicative or mixed mode began with "This guide," "This publication," 
or "The author," which as several evaluators noted is not considered the most 
appropriate format according to the processing manual. 

Selection of Abstract Type 



Of the 17 abstracts evaluated, 1 used an informative mode; 5 used an indicative 
mode; and 11 used a mixed mode. The choice of abstract type was considered 
appropriate in 12 cases. In the five cases where the choice of abstracts wa s 
considered inappropriate, the mixed mode had been used. 

Content : 

Most abstracts were felt to cover the contents of the documents quite well. 

Four abstracts were considered by evaluators to have major omissions, such as 
appendices, important sections, background material, and data tables. Four 
abstracts were felt to have been weighted i nappropr i a te 1 y in terms of the 
document. In two cases, the abstracts were felt to misrepresent the document 
ei'h r in tone or content. In one abstract an inaccuracy was noted. 

Several major problems revealed in many abstracts evaluated during the project 
did not occur with D 1 s abstracts at all, such as failure to provide a good 
picture of the general subject or purpose of the document. 

Length : 

0-50: 0 
51-100: 0 
101 - 150 : 1 
151 - 200 : 6 
2 r 3-250 : 7 

250+ 3 

The c 1 ca r i nghousc 1 s abstracts tend to exceed the 200-word limit. Several evaluators 
commented on this as a problem, and suggested means by which the abstracts could have 
remained under 200 words. In genera 1 the evaluators suggested extensive editing, 
which would be quite costly. 







C 1 c cl r inn house as Eva 1 ua tor 



The evaiuators both had processing responsibilities, eit'.her as editor or 
abstractor. One evaluator also had user services responsibilities, 
in total the evaluators awarded 8 excellent ratings, 7 good ratings, 10 fair 
ratings, and h poor ratings. The user services representative tended to assign 
higher ratings (7 of the 8 excellent ratings given). 

The c I ear i nghouse 1 s evaluators stressed the importance of including methodology and 
findings in abstracts of research studies. They frequently suggested adding details to 
short abstracts. Both were very critical of abstracts that failed to provide 
a true reflection of the contents or format of the document. Both underscored 
the importance of a readable and complete topic sentence. The user services 
representative tended to be more critical of poor writing style than the other 
eva I uator . 

The clearinghouse's evaluations were very complete, provided very useful suggestions, 
and in several eases provided alternative abstracts for very poor abstracts. 

in 6 of 2o cases the evaluators nil s i dent i f icd abstract type. 



E 



Number of E abstracts evaluated: 19 

Total number of evaluations completed on E abstracts: 30 

Number of abstractors whose work i * represented: 7 

Number of evaluations completed by E staff: 39 

Number of E e aluators: 9 

Over a 11 Rn l i ngs : 

1 excellent; 6 good; 8 fair; 9 marginal; and 6 poor 

Of the 30 evaluations completed, 7 provided ratings of excellent or good. 

Style a nd Vocabula r y_ 

For the most part, E's abstracts were rated highly in areas of style and 
vocabulary, including intelligibility, logical structure, correct grammar 
and spelling, understands i 1 i t;y to the non -spec ia 1 t st: , etc. In only two 
cases were the abstracts considered unclear or difficult to read. In only 
one abstract did evaluators point to terms which they thought should have been 
explained. Grammatical problems were noted in only one abstract. 

Select ion of Abs (met T y pj^ 

Of the 19 abstracts considered, 10 used an indicative mode, 2 used an informative 
mode, and / used a mixed mode. Choice of abstract type was considered appropriate 
in 14 cases and ! i nappropr i a to in 5 cases. Evaluators indicated that in several 
cases an informative mode would have been more appropriate than the indicative 
or mixed mode. 

Con tent 

V/hile at least 7 of the abstracts were felt to address all of the important, 
topics covered in the. documents, several problems were commonly cited in the 
abstracts receiving lower ratings. Evaluators pointed to major omissions in 
13 abstracts, indicating that the abstracts had failed to adequately deal 
with major concepts or important sections, conclusions, study results and 
methods. Two abstracts were critic I zed for failing to provide a good overview 
of the documents' subjects. Evaluators felt that two abstracts should have 
provided additional details. In one case, the evaluators felt the abstract 
misrepresented the. abstract. 

Length: 

0-50 words: 1 

51-100 words: 3 

101-190 words : 9 

151-200 words: 6 

Evaluators tended to be especially critical of what they considered important 
omissions in light of the shortness of many of the abstracts. 




Clearinghouse as Evaluator 



Five Clearinghouse staff members participated in the project, including three 
associate directors and two research associates. They awarded 13 excellent 
ratings* 17 good ratings, 4 fair ratings, and 1 marginal rating. Overall the 
ratings were quite high and few comments were included. 

Low ratings were assigned when major sections of a document were ignored in the 
abstract or when the abstract provided an inaccurate picture of the document's 
contents . 

Of the 35 evaluations, 10 mi s i dent i f i ed abstract type. 



F 



Number of F abstracts evaluated: 17 
Number of evaluations completed of F abstracts: 29 
Number of abstractors whose work is represented: 6 

Number of evaluations completed by F staff: A3 
Number of F evaluators: 6 



Qveral 1 Rati nqs : 

3 excellent; 14 good; 3 fair; 8 marginal; and 1 poor 



Of the 29 evaluations completed, 17 provided ratings of excellent or good. 

Selection of Abstract Type : 

Of the 17 abstracts evaluated, 6 used the Informative mode, 10 used the Indicative 
mode, and 1 used the mixed mode. Choice of abstract type was considered appropriate 
in 15 cases and inappropriate in 2 cases. 



Style and Vocabulary : 

While most of the abstracts were ra"ed quite highly in terms of style ana vocabulary, 
several were found to be difficult to read. Three abstracts were felt to use overly 
long and complex sentences. Evaluators commented that two abstracts were understand- 
able only after several readings. In one abstract the relationship of information 
appearing in parentheses to the. rest of the sentence was considered unclear. One 
evaluator commented that an abstract was very confusing. 



Content : 

Most of the abstracts were felt to cover all important topics of the^ 
documents. In three cases the evaluators pointed to what they considered 
to be major omissions, such as an imp rtant topic, research objectives, 
and implications. In two cases the evaluators felt the abstracts were 
overly generalized and to lack specific details. In three cases tne 
evaluators felt that the abstract was an inadequate representation of the 
document . 

Length : 



0-50 0 

51-100 1 

101-150 3 

151-200 13 

200 + 2 




Clearinghouse as Evaluator 



Six members of the Clearinghouse staff par t i c I pated in the project, including 
the Director, Associate Director, a user services specialist, an editor, and 
two information specialists. They provided a total of b 3 evaluations, and 
3 excellent ratings, 9 good ratings, 17 fair ratings, 6 marginal ratings, and 
7 poor ratings. 

These evaluations stressed the importance of ’'serving the user." They were 
critical of abstracts that did not provide comprehensive coverage; that 
failed to define technical terms; that failed to provide a good picture of 
the overall subject of a document or its format. They frequently found 
indicative abstracts to lack essential information. One evaluator, when 
considering selection of abstract type, was most concerned with whether the 
mode chosen "worked” in the given situation. 

In 9 of 29 cases, abstract type was mi s i dent i f i ed . 
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G 



Number of G abstracts evaluated: 15 
Number of evaluations completed of G abstracts: 33 
Number of abstractors whose work is represented: 5 

Number of evaluations completed by G satff: ^3 
Number of G evaluators: 4 



Qveral 1 Rat ings : 

9 excellent; 18 good; 3 fair; 3 marginal; no poor 

Of the 33 evaluations completed, 27 provided excellent or good ratings. 

This reflects one of the largest proportions of high ratings in the system. 



Style and Vocabulary 

Overall the abstracts received consistently high ratings in all areas of style 
and vocabulary, such as intel 1 ibi 1 ity, logical structure, correct spelling and 
grammar, understandab i 1 i ty to non-specialists, good topic sentences, etc. 

In four cases, the evaluators pointed to terms that they felt should have been 
explained in the abstract, linking this in some cases to an overuse of jargon. 
One abstract was felt to lack structure, and another to be unclear. 



Selection of Abstract Type 

Of the 15 abstracts evaluated, 7 used the informative mode, ^ used the indica- 
tive mode, and 4 used the mixed mode. In only 2 cases was the choice of abstract 
type considered inappropriate. 



Content 



Again, most of the abstracts were rated quite highly and felt to cover all of 
the important topics in the documents. The abstracts that received lower 
ratings had the following problems: evaluators felt that 3 abstracts could 

have provided additional details; major omissions were noted in two abstracts, 
e.g., the document's summary or conclusions; one abstract was criticized for 
failing to indicate the document's purpose. One abstract contained a factual 
error. 

Length : 



0-50 v;ords: 0 
51 -1 00 words: 2 
101-150 words: 3 
151 -200 words: k 
200+ words: 5 



Clearinghouse as Eva 1 ua t o r 



Four staff members participated in the project, two abstractors, one editor, and 
the processing coordinator. They awarded 2 excellent ratings, 11 good ratings, 

10 fair ratings, 10 marginal ratings, and 10 poor ratings. This reflects some of 
the lowest ratings overall of any group of evaluators. 

The evaluators were highly critical of poorly written abstracts, especially those 
which used long and overly complex sentences. Their evaluations stressed the 
importance of comprehensiveness, appropriate weighting of the abstracts, and the 
importance of providing an accurate representation of a document and its contents- 

ln 9 of ^3 abstracts the evaluators mi s i dent i f i ed abstract type. 

Comments on a few of the evaluations implied that a document of questionable 
quality did not need extensive treatment in the abstract. 



3EST COPY AVAILABLE 



Number of H abstracts evaluated: 2k 
Number of evaluations of H abstracts: 37 
Number of abstractors whose work is represented: 2 

Number of evaluations completed by H staff: 33 
Number of H evaluators: 3 



Ove ra 1 1 ratings : 

11 excellent; 15 good; ? fair; no marginal; no poor 

Of the 37 evaluations completed, 26 proviaed excellent or good ratings. 
Style and Vocabulary 



Most of the abstracts were rated highly in all areas of style and vocabulary. 
One abstract was felt to contain grammatical problems; 2 to be difficult to 
read; 1 to be redundant in its word usage; and 1 to be confusing. 



Selection of Abstract Ty pe 

Of the 2k abstracts evaluated, 8 used an indicative mode, 7 used an informative 
mode, and 9 used a mixed mode. In all cases the choice of abstract type was 

considered appropriate by the evaluators. 

Content 



Generally the abstracts were also felt to cover the major topics of the 
documents. In one case major omissions were noted; 1 abstract was felt not 
to provide an adequate statement of the document's purpose; 1 to have a 
weighting problem; and 1 to misrepresent the document. 

Length : 



0“50 words: 0 
5 1 - 1 00 words : 0 
101-150 words: 0 
151-200 words : 1 8 
200 + words: 8 



Clearinghouse as Eval uator 



Three staff members participated in the evaluations, representing processing, 
user services, and management. They awarded no excellent ratings, 6 good 
ratings, 8 fair ratings, 11 marginal ratings, and 8 poor ratings. 

The evaluators stressed the importance of abstracts providing the most comprehensive 
coverage possible. They tended to feel that in most cases abstracts under 175 words 
in length could have provided additional details or examples. They seemed to 
consider problems with abstract content, especially major omissions, misrepresenta- 
tions, and failure to provide a good picture of the overall purpose or subject of a 
document to be major weaknesses. They considered problems with style and vocabulary 
to be secondary to subject coverage. 

In 1 of 33 abstracts evaluated, abstract type was m i s i den t i f i cd * 
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M 



Number of different M abstracts evaluated: 12 

Total number of evaluations of M abstracts completed: 22 

Number of abstractors whose work is represented: 5 

Number of evaluations completed by M staff: 31 

Number of M evaluations: ' 2 

Qveral ] Rat i ngs : 

3 excellent:; 3 good; 10 fair; 6 marginal; no poor 

Of the 22 evaluations completed, 6 provided overall ratings of good or excellent. 
Selection of Abstract Type : 

Of the 12 abstracts considered, 0 used the informative mode, 5 used the indicative 
mode, and 7 used the mixed mode. Choice of abstract type was considered appropriate 
for 7 of the abstracts. in the five cases where abstract mode was considered 
inappropriate, the evaluators tended to feel that an informative mode would have 
been more effective than either the indicative or mixed. 

Style and Vocabulary: 



Generally the abstracts were rated quite highly in the areas of style and 
vocabulary. Evaluators tended to find the abstracts intelligible, to use 
correct grammar and spelling, to be understandable to the non-specialist, 
and to provide good topic sentences. In three cases the evaluators noted 
terms that they felt should have been explained in the abstracts. In two 
cases, the abstracts were felt to lack a logical structure. Three abstracts 
were thought to be difficult to follow. 

Content: 



In most cases the evaluators tended to feel that the abstracts were too short 
to adequately cover the contents of the documents. Of the 22 evaluations 
completed 12 cited major omissions of the abstracts, such as findings, methodology, 
major topics, or important sections. In one case the evaluator felt the abstract 
failed to provide a good representation of the overall subject of the document. 
Inaccuracies were pointed to in one abstract. 

Length : 



0-50 words: 1 

51-100 words: 1 

101-150 words : 7 

151-200 words : 2 

200+ words: 1 

As most of the abstracts in the sample used less than 150 words, the evaluators 
were very critical of what they considered to be major omissions in the abstracts. 

Clearinghouse as Evaluator 

Two Clearinghouse staff members participated in the project, the Assistant 
Director and the User Services Coordinator. They assigned 3 excellent ratings, 

18 good ratings, ^ fair ratings, 5 marginal ratings, and k poor ratings. 

Several of the higher ratings were somewhat surprising as the accompanying () 

comments pointed to major omissions, failure of the abstract to reflect the 
documents tone, overly general abstracts, and such complicated sentences that 
it took the evaluators several readings to understand the abstract. 



The evaluators stressed the Importance of comprehensive coverage of all 
important topics in the document, the value of references to survey 
instrument's and appendices, and weighting abstracts to reflect the documents 1 
emphases. The evaluators mi 5 ident i f ied the abstract type in 7 of 31 
eval uat ions. 



N 



Number of different U abstracts evaluated: 

Number of evaluations of N abstracts completed: 
Number of abstractors whose work is represented: 

Number of evaluations completed by N staff: 
Number of N evaluators: 

Overal 1 Rati ngs: 



3 excellent; Wi good; 12 fair; 6 marginal; and 2 poor 

Of the 37 evaluations completed, 17 provided ratings of excellent or good. 
Several of the abstracts were rated as fair strictly on the basis of choice 
of abstract type, with the evaluations revealing that the abstracts were 
comprehensive and readable. 

Selection of Abstract Type : 

Of the 27 abstracts considered, 2 used an informative mode, 6 used an 
indicative mode, and IS* used a mixed mode. Choice of abstract type was 
considered appropriate in 17 cases. In the 10 cases where abstract type 
was considered i nappropr i ate , evaluators tended to favor the informative 
mode over the mixed mode. 

Style and Vocabulary : 

Generally the abstracts were rated highly in areas of style and vocabulary. 
They were considered to be intelligible, to use correct grammar and spelling, 
to be understandable to the non-specialist, and to explain new or technical 
terms. In only two cases were the abstracts considered difficult to read. 
Only one lacked a logical structure, and only two were felt to use long and 
overly complicated sentences. 

Content : 

Most abstracts were felt to cover the content of the documents well. In 
those abstracts receiving lower ratings, the most common criticism was 
that important topics or facts had been omitted (eight evaluators pointed 
to what were considered important omissions). One abstract was felt to 
lack sufficient detail. Two were criticized for not providing a good idea 
of the overal 1 subject of the document. 

Length: 

0-50 words: 0 

50-100 words: 0 

1 01 -1 50 words : 3 
1 50-200 words : 1 2 
200+ words: 12 

In several cases the evaluators were concerned that the abstracts were 
overly long. In two cases they indicated that length was attributable to 
repeating information. 



4 
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Clearinghouse as Evaluator 



Five Clearinghouse staff members participated in the project, the associate 
director, assistant director, two staff members whose responsibilities include 
abstracting, and a user serv i ces/publ i cat ions representative. 

The evaluators assigned 7 excellent ratings, 15 good ratings, 8 fair ratings, 
and 3 marginal ratings. 

The evaluators stressed the importance of providing a good idea of the document 
as a whole, of appropriate weighting, and of comprehens i ve coverage. They 
were very critical of very short abstracts providing superficial coverage. 

In several cases they felt that purely indicative abstracts had failed to provide 
enough information about the subject of a document. The importance of including 
author’s names in collections for on-line bibliographic retrieval was stressed. 

In 3 of 26 evaluations, the abstract type was mi 5 ident i f i ed. 




0 



Number of 0 abstracts evaluated: 16 
Number of evaluations completed of 0 abstracts: 26 
Number of abstractors whose work is represented: 3 
Number of evaluations completed by o' staff: 30 
Number of 0 evaluators: 2 



Ove ra 1 1 rat i ngs : 

3 excellent; 10 good; 7 fair; 5 marginal; 1 poor 

Of the 26 evaluations completcd } half provided ratings of excellent or good. 

Se 1 ect ion of Abstract Tvp e 

Of the 16 abstracts evaluated, | used an informative mode; 8 used an Indicative 
mode; and 6 used a mixed mode. The choice of abstract type was considered 
appropriate in 12 cases. In most cases where the choice of abstract type was 
considered i nappropr iate for the document, the indicative mode had been used 
when evaluators felt an informative mode had been called for, e.g., for research 
studies . 

Style and Vocabulary 

Generally, the abstracts were rated fairly highly witn regard to style and 
vocabulary. The most commonly identified problems we-c failure to explain 
terms or concepts ( 5 cases ); use of overly long and complicated sentences 
(A cases); grammatical problems (3 cases); and lack o 7 clarity, due in one 
case to problems using the mixed mode. 

Content : 

While at least half of the abstracts evaluated were felt to cover all of the 
important topics of the documents, the abstracts which received lower ratings 
evidenced common problems with content. Eight abstracts were felt to have 
major omissions, such as important sections of the document, study results and 
methodology, and important appendices. The evaluators felt that six abstracts 
could have provided additional detail. Three abstracts were felt to misrepresent 
the contents or format of a document, and another three were considered to provide 
an inadequate overview of the general subject of the document. Two abstracts 
contained i naccurac i es . 

Len gth : 



0-50 words: 0 

51-100 words: 2 

101-150 words : 3 

1 pi -200 words : 1 0 
201-250 words: 1 

Most abstracts ranged between 100 and 175 words. Most evaluators who felt 
that additional devils could have been provided were commenting upon fairly 
short abstracts. 
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Cj ea r j nghousc as Evalu a tor 

One evaluator was the Clearinghouse's Assistant Director; the other was an 
abstractor. The evaluators awarded 8 excellent ratings; 12 good ratings; 

6 fair ratings; h marginal ratings; and no poor ratings. 

The evaluators tended to rate the abstracts rather highly and to make few 
comments. Surprising high ratings v/c rc given to abstracts that the evaluators 
seemed to feel had major flaws, such as failure to provide an accurate representation 
of t lie document, lack of sufficient detail, and major problems in weighting. 

Both evaluators expressed concern when abstracts repeated information already 
provided in the titie or other cataloging fields. 

Abstract type was mi s i dent i f i ed in 6 out of 27 abstracts evaluated. 
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Total number of different P abstracts evaluated: 21 
Total number of evaluations completed of P abstracts: 31 
Number of P abstractors whose work is represented: 3 

Number of evaluations completed by P staff: 27 
Number of P evaluators: 3 



Overa 1 1 rat ? nq s : 

5 excellent; 9 good; 8 fair; 5 marginal; and h poor. 

Of the 31 evaluations completed, 1 A provided ratings of good or excellent. 

Selection of Abstra c t Ty po: 

Of the 21 abstracts evaluated, h used an informative mode, 8 used an 
indicative mode, and 9 used a mixed mode. In mo3t cases the ct.oice of 
abstract type was considered appropriate. In the four cases where the 
type was considered unsuitable for the document, the evaluators felt that 
the informative mode would have been more effective than the indicative or 
mixed mode. 

Style and Vocabulary 

For the most, part the abstracts w err* rated highly in areas of style and 
vocabulary. Seven abstracts were considered unclear or difficult to read; 
this was due to two cases to overly long and complex sentences. Four 
abstracts contained terms or concepts that the evaluators thought should 
have been explained. 

In two cases the mixed mode was felt to have been used i nef feet i ve I y by 
switching between informative and indicative too often for clarity. 

Content : 

While most of the abstracts that received high ratings covered all of the important 
topics of the documents, evaluators felt that several other abstracts had 
major content problems. In ten abstracts, evaluators noted what were considered 
major omissions, such as important sections, appendices, and, in the cases of 
research studies, information on findings and results. In two cases the evaluators 
felt that additional detail and information could have been provided. Three 
abstracts were felt to contain inaccuracies. 

Len gth : 

0-50 words : 0 

51-100 words: 2 

101-150 words: 

1 50-200 words : 3 

200:- words: 2 

As the abstracts tended to use between 100 and 150 words, the evaluators were 

especially critical of omissions and overly general treatments. 
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Three staff members participated in the abstract project: two of them were 

abstractors and one was the Assistant Director. The evaluators awarded 
3 excel lent ratings; 9 good nings; 10 fair rating*; k margins I ratings; and 
1 poo r rating. 

The evaluators tended t.o comment that abstract length should he reflective of 
document length. Long abstracts of short documents were felt to misrepresent 
the document; lengthy documents were fell to require, long abstracts, 

The evaluators underscored the importance of comprehensiveness, good representation 
of the overall subject of a document, clarity, and definitions of technical terms. 
All evaluators felt in some, cases Lhat additional detail should have been provided. 



In 3 out of 2k abstracts evaluated, the abstract type was mi s 1 den t i f icd . 
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Total number of Q. abstracts evaluated: 2b 
Total number of evaluations completed of Q, abstracts: ’ 36 
Number of abstractors whose work is represented: b 

Number of evaluations completed by Q. staff: 35 
Number of Q. evaluators: 6 



Overa 11 Ratings: 

3 excellent; 12 good; 9 fair; b marginal; 2 poor 

Fifteen of the 36 evaluations completed for Q. 1 s abstracts reflected ratings of 
excellent or good. Several of these evaluations included comments on the 
exceptional quality of the abstracts. 

Selection of Abstract Type: 



Of the 2b abstracts evaluated, 13 used the informative mode, 3 used the indicative 
mode, and 8 used the mixed mode. The evaluators considered the choice of mode 
appropriate in 20 cases. The four cases where the mode was considered inappropriate 
for the document used the mixed mode. The evaluations indicated that the mixed mode 
was used in an unclear and ineffective manner in two cases. 

Style and Vocabulary : 

In most cases the style and vocabulary of the abstracts were rated as being of 
high quality. In two cases the abstract was considered unclear. In three cases 
grammatical problems with subj ect -verb agreement or tense problems were noted. 

Four abstracts were felt to overuse jargon or to use technical terms without 
explaining them; in one case an evaluator felt that the abstract was at fault for 
explaining concepts without using the author's language to name them. 

Content : 



In most cases the abstracts were found to cover all of the important topics 
of the documents. Four evaluators pointed to cases in which the abstract 
could have provided additional details. Seven evaluators noted what they felt 
were major omissions in the abstracts being evaluated. In two cases the abstracts 
were evaluated as not providing a good representation of the overall subject of 
the document. 

Four abstracts were found to contain inaccuracies or to misrepresent sections 
of the document. In one of these cases, evaluators felt that the abstract revealed 
abstractor bias. 

Length : 



0“ 50 word s : 0 
51-100 word s : 2 
101-150 wo rds : 5 
151-200 words : 1 5 
200+ words: 2 



G s abstracts tended to come very close to the 200-word limit. In most cases 
where the evaluators felt that additional detail could have been added or major 
omissions had occurred the abstracts were under J-50. words. : 




Clearinghouse as Evaluator ^ 



The evaluators were all processing staff. In total, they awarded 2 excellent 
ratings, 3 good ratings, 15 fair ratings, 9 marginal ratings, and no poor 
rat i ngs . 

They tended to be highly critical of omissions. One evaluator was equally concerned 
with the style and readability of the abstracts as with their content. Several 
were very concerned v/ith abstracts well over 200 words, reflecting a Clearinghouse 
understanding that 200 words were the absolute maximum. Several indicated that ft 
was very important for the abstracts to provide an overall reflection of the purpose 
and format of the document. 



In four of 28 cases, evaluators mi s i dent i f i ed abstract type. 
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Total number of different R abstracts evaluated; 27 
Total number of evaluations completed of R abstracts; 4l 
Number of abstractors whose work is represented: 4 

Number of evaluations completed by R staff; 34 
Number of R evaluators: 6 



Qvera 1 1 Ratings; 



4 excellent; 18 good; 9 fair; 8 marginal; and 2 poor 



Of the 41 evaluations completed, 22 provided ratings of good or excellent. 
Se 1 ect ? on of Abstract Type: 



Of the 27 abstracts evaluated, 7 used the informative mode, 8 used the indicative 
mode, and 2 used the mixed mode. In all but two cases, the choice of abstract 
type was considered appropriate. 



Style and Vocabulary 

In general the abstracts received fairly high ratings in the areas of style and 
vocabulary. Structural or language usage problems resulted in 11 abstracts being 
rated as unclear or unintelligible. Another commonly cited problem was the use 
of unexplained terms or jargon--eval uators noted this problem in 6 abstracts. In 
only two cases were grammatical problems noted. 



Content : 

Over half of the abstracts covered all of the important topics of the documents. 
Low ratings for ten of the abstracts were due to what the evaluators considered 
to be important omissions, such as the results and methodology of research 
studies, important sections, or background information. Three abstracts were 
considered to lack sufficient detail; w L » 1 e five abstracts were criticized for 
not adequately representing the overall oubject of a document. Evaluators 
pointed to inaccuracies or misrepresentation of the document in 4 abstracts. 

Length : 



0-50 words: 0 
51-100 words : 3 
101-150 words : 7 
151-200 words : 1 4 
200-*- words: 3 

Most of the absuracts were close to 200 words. 
C 1 ear i nqhouse as Evaluator 



Six staff members participate] in the abstract project. While most of the evaluators 
were abstractors, management and user services were also represented. 

The evaluators awarded a total of 4 excellent ratings, 7 good ratings, 13 fair 
ratings, 5 marginal ratings, and 5 poor ratings. 

The evaluators tended to give equal weight to style, content, and choice of abstract 
type. Problems recognized in any of these areas were likely to result in low ratings. 
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The evaluators stressed the importance of readability, accurate representation 
of the contents or format of a document, weighing of the abstract in the same 
manner as the document, and appropriate use of the mixed mode. in two cases, 
in-house guidelines for abstracts seemed to be confused with the guidelines 
provided in the processing manual. 

tn A of 30 cases, the evaluators m i s i den t i f i ed abstract type. 








s 



Number of different S abstracts evaluated: 

Total number of evaluations completed of S abstracts: 

Number of abstractors whose work is represented: 

Number of evaluations completed by S staff: 

Number of S evaluators: 

Ove rail ra t i ngs : 

2 excellent; 8 good; 7 fair; 3 marginal; and 4 poor 
Of the 2k evaluations completed, 10 provided ratings of excellent or good. 
Selection of Abstract Type : 

Of the 15 abstracts evaluated, 5 used the informative mode, 9 used the indicative 
mode, and 1 used the mixed mode. In all cases, the choice of abstract type was 
considered appropriate. Only one other Clearinghouse was considered to have used 
the appropriate abstract type so consistently. 

Style and Vocabulary 

Generally S*s abstracts v/ere rated quite highly in all areas of style and 
vocabulary. (n only two cases did the evaluators find abstracts difficult 
to follow and in only three cases did they comment on technical terms that 
should have been explained. In most other uases the abstracts were considered 
very well written. 

Content : 

Six of the abstracts were felt to have covered all of the important topics in the 
documents. Among those abstracts that received lower ratings, the most frequent 
criticisms were that the abstracts were too general (6 cases) or had major 
omissions (5 cases), such as study purposes or results, implications, and 
authors/titles in the case of a collection. In one instance, the evaluators 
found the abstract to be misleading and, in one case; to provide an inadequate 
representation of the overall subject, of the document. 

Length : 

0-50 words: 0 

51-100 words : k 

I 01 -1 50 words : 3 
1 51 -2 00 word s ; 8 

Given that most of the abstracts evaluated were less than 175 words long, the 
evaluators tended to be especially critical of abstracts that they felt provided 
superficial coverage or that lacked details and examples that could have clarified 
the subject of the document. 

Clearinghouse as Evaluator : 

Only one evaluator, the Assistant Dii jctor, participated in the abstract 
project. She assigned 2 excellent ratings; 2 excellent-good ratings; 13 
good ratings; 2 good-fair ratings; 5 fair ratings; 5 fa i r-ma rg i na 1 ratings; 

6 marginal ratings; and 2 poor ratings. 
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S's evaluations revealed a preference for the informative mode, with 
comments indicating that in many cases the indicative mode fails to 
provide a clear indication of the subject of a document. in several 
evaluations, comments were made indicating that an abstract’s length 
should he reflective of the document’s length. Most evaluations stressed 
the importance of weighting in an abstract, with low rating; given to 
abstracts that omitted sections or under st ressed sections that seemed to 
be of significance in the document. The evaluator underscored the importance 
of indicating the presence of appendices, especially those containing survey 
instruments. The evaluator pointed out grammatical flaws and stylistic 
problems in the abstracts, yet understood that abstracts did not merit low 
ratings simply on the basis of these problems. Clarity, comprehensive 
coverage, and proper choice of abstract type were considered critical evaluation 
cr i ter i a . 

S was the only clearinghouse to have identified abstract type correctly in all 
cases . 
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Number of different T abstracts evaluated: k] 
Total number of evaluations completed of T abstracts: 62 
Number of abstractors wh6se work is represented: 3 

Number of evaluations completed by T staff: 38 
Number of T evaluators: ? 



Qvera 11 ratings : 

7 excellent; 20 good; 18 fair; 15 marginal; and 2 poor 

Of the 62 evaluations completed, 27 provided ratings of excellent or good. 



Selection of Abstract Type : 



in the sample of ^1 abstracts, 8 
and 13 mixed abstracts were used 
type was appropriate in 3A cases 



informative abstracts, 20 indicative abstracts, 
Evaluators felt that the choice of abstract 
and inappropr i a te in 7 cases. 



Style and Vocabulary 



For the most part, T's abstracts were rated highly in the areas of style and 
vocabulary. They were generally considered to be understandable, to explain 
new and technical terms, and to provide good topic sentences. In 6 of A2 
abstracts, the evaluators pointed to what they considered to be serious 
grammatical errors, such as lack of subject-verb agreement, mixed or incorrect 
tenses, or complicated d i f f i cu 1 t -to-fol low sentences. In only two cases did 
evaluators recommend a more logical structure. 

Content 



Wh i 1 e 27 of the abstracts evaluated were felt to have covered all of the 
important topics in the documents, several problems were commonly recognized 
in the abstracts receiving low ratings. 

Authorship of specific sections and articles tends not to be 
indicated. (This was the case in three of the three abstracts 
where authorship statements would have been appropr iate , ) 

Five abstracts were felt to be unreflective of the document. 

Evaluators indicated that 6 abstracts were too general; that 
they lacked specific details that would have been useful in 
understanding the contents of the documents. 

Abstracts of research and evaluation studies tended to be weak 
with respect to methodology and findings. (Abstracts of ? 
research studies were evaluated, methodology was considered to 
have been inadequately covered in 8 abstracts and findings 
inadequately covered in ^i.) 

In cases where several related documents were being processed 
in one issue of R 1 E , the abstracts tried to relate the documents 
in hand with other documents in a manner that was considered 
confusing. Several evaluators felt that the abstracts should 
restrict themselves to the document being abstracted and allow 
the note field to guide the user to other related documents. 

BEST COPY AVAILABLE 2 t 



Length 



0-50 words: 0 
5 1 - 1 00 words ; 1 
101-150 words : 1 1 
150-200 words: 20 
201 -250 words: 5 
250+ words: 5 
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Evaluator 



Three members of the Clearinghouse staff participated In the evaluations: 
an abstractor, the processing manager/editor, and acquisitions coordinator, 
They assigned 2 excellent ratings, 15 good ratings, 8 fair ratings, 5 marginal 
ratings, and 8 poor ratings. 

The acquisitions coordinator assigned consistently high ratings. The other 
two evaluators stressed the importance of abstracts being clearly written 
and that they define terms that might not be easily understood. Both 
evaluated abstracts in terms of their usefulness to the abstract user. 

While the evaluators pointed to omissions, they tended to weigh these 
omissions in terms of their importance to the document. They were concerned 
with comprehensiveness and the use of the appropriate abstract type. 



in 6 of 29 cases, evaluators m i s i den t i f i ed abstract type. 
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System as a Whole 




Overall Ratings : 

Of the 526 evaluations completed, 67 excellent ratings, 188 good ratings, 150 
fair ratings, 91 marginal ratings, and 30 poor ratings were assigned. As noted 
in the introduction, these ratings cannot be taken at face value. They do 
provide, however, a fairly good overall look at quality in the system. If the 
evaluations were redone, methods instituted to insure consistency, standards 
established for assigning each rating, the ratings would certainly change. 

Several excellent ratings would become good, fair or perhaps even marginal; 
several poor ratings would become marginal, fair, or perhaps even good. Overall, 
however, it is likely that the pattern would remain much the same. 

As can be seen, a little loss than half of the evaluations providing ratings of 
good or excellent. Given that a certain number of the fair evaluations were 
assigned on the basis of rather minor problems, it would probably be reasonable 
to assume that between 175 arid 200 of the abstracts evaluated were acceptable by 
ERIC standards. 

Selection of Abstract Type 

Selection and use of abstract type remains a fairly significant problem. 

Evaluators misidentified abstract type on 105 of 526 evaluations. This, of 
course, is not a critical problem when the evaluators were user services or 
management personnel. It does however reflect a potentially serious problem 
in cases where these evaluators were abstractors themselves. If abstractors 
are unable to tell one mode from another, chances are much greater that they 
will mix modes improperly or choose an inappropriate abstract type. 

Indeed, the evaluators felt that the choice of abstract type had been inappropriate 
for 66 of 349 evaluations. 

The evaluations revealed that each abstract type has strengths and potential 
weaknesses. The informative abstract has the greatest potential for providing 
information to the user. "However, the effectiveness of this abstract type is 
based on the reader's ability to infer what the. document contains from what 
the abstract says. In several cases, the evaluators felt that the informative 
abstract had failed to give the reader a good idea of what the document contained. 

The medicative abstract, on the other hand, allows the abstractor to tell the user 
exactly what the document contains. Used ineffectively, however, the abstractor 
can fail to tell the user what the document is about. To indicate that "study 
methodology is detailed" is singularly useless if the reader has no idea of what 
the study sought to determine and how it went about doing so. To say that 
various aspects of a reading program are described is very unhelpful if the 
abstract fails to give an idea of how this reading program differs from countless 
others. In most cases, where the indicative mode was considered inappropriate, 
it had failed to give an good picture of the document's subject. 

The mixed mode abstract has enormous potential for providing a great deal of 
information and "indicating to the user exactly what the document contains. It 
must however be used very carefully. Abstracts which switch from the informative 
to indicative modes throughout are extremely difficult to read and make it 
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hard to determine both what the document contains and what it is about. In 
several cases, the evaluators agreed that the mixed mode was appropriate, but 
felt that it had been used improperly. In some cases, the evaluators tended to 
indicate that certain Clearinghouses over-rely upon the mixed mode. 

Style and Vocabulary 

The ERIC Processing Manual adjures the Clearinghouses to use good prose style 
in their abstracts. It recommends the use of varied sentence patterns, varied 
in terms of length and voice. It stresses the importance of good topic sentences 
and logical structure, and cautions abstractors about readability and understand- 
abi 1 i ty . 

These standards were reflected in the abstract evaluation form. Seme evaluators 
tended to feel that these standards of writing style' were unimportant, feeling 
that varied sentence patterns were secondary to information density or that 
abstractors must assume that the users of the system have enough knowledge of 
a particular field to understand its terminology and jargon. Other evaluators 
felt that good writing style is critical to good abstracting. 

The most common problem identified was that the abstracts tended to use the same 
sentence patterns, but most evaluators agreed that this wasn't a serious problem. 
Within this category of problem, the excessive and monotonous use of the passive 
voice was considered the most serious. 

Probably the most serious problem, and one that occurred with some frequency, was 
the lack of readability of certain abstracts. In many cases, the evaluators found 
an abstract "intelligible" only after several readings. In some instances this 
was due to very long and complicated sentences. In some cases, examples were 
provided in parentheses and the evaluators were unable to tell to what these 
examples referred. Unclear or indefinite anti cedents of pronouns was another 
problem that detracted from the understandabi 1 i ty of abstracts. 

Several spelling errors ware identified in the abstracts. It is impossible to 
tell, however, whether these usually typographical errors were the responsibility 
of the abstractor, the OCR scanner, or the type-setter. Spelling errors were 
considered a major problem when the names of individuals or instruments were 
incorrectly spelled. 

Certain evaluators were able to identify even minor grammatical problems, 
such as the proper use of punctuation and hyphenation. Most were either not 
as knowledgeable or not as concerned with these minor problems. However, 
many serious grammatical problems, which detract from the overall image of 
quality of the ERIC system, were also identified. These included lack of 
verbs, lack of subject-verb agreement, and inconsistent use of tenses. 

Content 



The most serious content problem, and the one occurring with greatest frequency, 
was the omission of important sections or topics of a document. Another common 
and serious problem was the misrepresentation of a document. In several cases, 
the evaluators indicated that they had been lead to expect something very different 
from a document than what they found when they went to the document itself. A 
fairly rare, but serious problem was that of inaccuracies. 
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Evaluators frequently felt that short abstracts, even when providing a good general 
description of a document, could have provided many additional and helpful details 
by using the entire 200 words allowed. Another commonly cited problem was that of 
weighting. In some cases, this was considered a serious problem when the emphases 
of the abstract tended to give a distorted picture of the document. 

Host evaluators tended to agree that if a document contains a survey instrument 
or extensive data tables, these items should be noted in the abstract. 

Certain evaluators stressed the importance of including the names of authors of 
sections or articles within a document in the abstract, pointing out the value 
of this information for on-line retrieval. 

L ength : 

Short abstracts of less than 150 words are very common in the system, especially 
among certain Clearinghouses. Most evaluators indicated that short abstracts 
were acceptable, as long as all important aspects of a document were covered. 

They were very critical of short abstracts that omitted sections, glossed over 
contents, or failed to give a good indication of the document's subject. Some 
evaluators, generally from Clearinghouses that consistently approach or exceed 
the 200-word limit on length, felt that in most cases abstracts should use the 
entire space allotted to provide as much information as possible. 

Very long abstracts (i.e., exceeding 250 words) were also fairly common among 
a few Clearinghouses. Some evaluators saw this as a problem and suggested that 
careful editing would have reduced the length of the abstracts. When considering 
long abstracts, it must be realized that a substantial amount of time would be 
needed to reduce the lengths of these abstracts without affecting the amount of 
information provided. 

As several evaluators noted, length per se is not an especially useful criterion 
for judging abstracts. They felt that the number of words is not as important as 
how effectively these words are used. 



Recommendations 



Based on the findings of the Abstract Evaluation Project, the following recommendations 
are made: 

A systemwide workshop for abstracting and editing staff should be 
convened to accomplish the following purposes: 

1) increase understanding of the differences between abstract types 
and the documents for which each typo is most appropriate; 

2) help Clearinghouses use the mixed inode effectively and only when 
it is appropriate; 

3) promote the practice of editing in the Clearinghouses to lessen the 
number of grammatical errors, spelling errors, and factual errors, 
as well as to insure the readability of abstracts. 

4) review the guidelines established in the ERIC Processing Man ual for 
informational content for specific types of documents. 

5) achieve a greater consensus among the Clearinghouses as to the 
purpose of ERIC abstracts. 

ERIC personnel, including Central ERIC and Clearinghouse staff members 
and the Steering Committee, should consider methods for periodic or' on-going 
review of abstract quality. 



Central ERIC should investigate the relationship between unit cost and 
abstract quality, taking into account abstractor salaries, expectations of 
abstractor productivity, and amount of time expended on editing. A determination 
should be made as to wiiat level of quality is expected systemwide and how much 
more time and money will be needed to achieve this level of quality. Abstractors 
who are expected to produce an abstract in 15 or 20 minutes will never ■ hi eve 
the quality of those who are permitted an hour or more to complete an a Lm iract. 
Abstracts that never go throgh an editing process will always contain more errors 
in writing and content than those that undergo a thorough editing. 
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R esu lts of the Fjjot Study 

In October we sent out three abstracts to be evaluated by each C 1 ca r i ncjhouse 
as a pilot test of the abstract evaluation form and to determine whether the 
form could be used consistently by all Clearinghouses and evaluators. 

We discovered: 

(1) though most people found the form easy to use, a significant number 
had difficulty determining publication type. The pilot also revealed 
a need for more specific questions related to overall abstract 
quality to draw more substantive responses from the evaluators. 

(?) there was very little consistency among the evaluators. While all 
three abstracts had very serious problems, all three received 
overall evaluations ranging from Good to Poor. Serious problems, 
such as the inclusion of misinformation, the failure to reflect the 
purpose of a document, and failure to g i vt consideration to a major 
portion of the document, were ignored by many Clearinghouses. 

We responded to the C 1 ea r i nghouses with very specific letters pointing out 
the failings of each of the abstracts and of each of the evaluations. We 
asked that the evaluations be performed more thoughtfully and conscientiously. 

We revised’ the form to alleviate the problems experienced by those evaluators 
who had difficulty in completing the p i lot -tested vers ion , and mailed out new 
forms, a revised set of i ns t ruct i ons , an explanatory cover letter, and an 
assignment of 16 absirocts to be evaluated on December 7 requesting their return 
by January , 1992. 

Tentative Results of the First Series of Evaluations 

The January ^th deadline was very difficult for most Clearinghouses to meet due 
to the holidays and the fact that the microfiche shipments from EDRS were delayed 
by three weeks; most C 1 en r i nghou se s had not received their November 19S1 microfiche 
by December iHh. 

By January 13, 1 93T; we had received complete sets of evaluations from 1J 
Clearinghouse: 'CS, EA, EC, HE, JC, PS, RC , SO, SP , TM, UD, CG OCE), one partial 
set of evaluations from EL, and a letter from SE indicating the reasons for the 
delay In their response. 

The evaluations we received reveal that: 

(l) some C 1 ea r i nghous es seem to have established their own standards of 

abstract wr i t ing, apart from the Processing Manual, which they are applying 
in thei r eivaluat ions. 




(2) great inconsistency remains in the evaluations. Many evaluators are having 
difficulty recognizing good and poor abstracts. Good abstracts were . \ - 
rated as poor not because they did not cover the content of the documents 
adequately, but because they used a complicated writing style or because 
they did not follow C 1 ea r i nghousc - spec i f i c guidelines in their approach. 

Poor abstracts were given good or excellent ratings, perhaps because they 
are easy to read, perhaps because the evaluator did not examine the 
document closely enough. 

(3) some evaluators, especially those who are involved in Clearinghouse 
activities other than abstracting or editing, are following the guidelines 
of the Processing Manual too closely. They quote the Processing Manual* s 
overall guidance regarding a document type, rather than considering whether 
specific document characteristics may make following a specific guideline 
inappropriate for a particular document. 

(M some evaluators are not aware of the guidelines at all. For example, 
one 215-word abstract '/as rated down as being too short. 

These problems with the evaluations make it impossible to send the abstract- 
originating Clearinghouses back the evaluations as originally intended. The 
well-done evaluations providing thoughtful criticism cannot be mixed in with 
those that were completed thoughtlessly or in ignorance. The Clearinghouses 
receiving such a mixture of critiques would be justified in dismissing the 
validity of the whole project. 

We propose the following alteration in the research design: 

Instead of selecting a large sample of abstracts to be evaluated only 
once, we will select a smaller sample of abstracts to be evaluated by 
four Clearinghouses. Our Clearinghouse will collate the four evaluations 
and provide a composite of the responses. We will send back the evaluations 
with comments on the appropriateness of the content and comments to the 
evaluators. We will not provide feedback to the originating Clearinghouse 
until the evaluations are reliable. We expect that the effort used to 
educate the evaluators about the qualities of good and bad abstracts and 
about the processing manual will have a beneficial effect on abstract 
quality as well, as most of the evaluators are involved in either 
abstracting or editing of abstracts. 

Because many Clearinghouses are finding the 16-abstract assignments 
overly burdensome, and because our Clearinghouse will be responsible 
for providing a composite evaluation of each of the abstracts selected 
for evaluations, we are going to reduce Lhe size:of the evaluation assign- 
ments to 12 absts. a month. We anticipate raising this number back 
to the original 16 for the last month of the project in order to have 
a sufficiently large and representative sample for the final evaluation 
of the abstracts, scheduled for July 1932. 
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ABSTRACT EVALUATION PROJEC T 

Interim Report ~2 
April 14, 1982 

Tentative Results of the Second Series of Evaluations 



On February 2, 1982 the Clearinghouse mailed out a letter to the Directors 
of the Clearinghouses, informing them of the problems of consistency and 
differing standards revealed by the first series of evaluations; advising 
them of certain methodological changes; and assigning each Clearinghouse 
12 abstracts from the January issue of Resources in Education to be evaluated 
and returned by March 1st. 

By the third week in March, all but one Clearinghouse had returned comnleted 
evaluations for their assignments and the Documents Coordinator began assessing 
the results and preparing feedback. This involved: 

(1) preparing composite evaluations from the 3-4 submitted evaluations. 
The: composites include all of the valid comments from .the 

evaL.utors and attempts in very constructive and positive terms to 
provide an evaluation which will be useful to the originating 
abstract. Samples of these composite evaluations are enclosed. 

While it was our original intention to use these composite evalua- 
tions solely for the purpose of aligning the standards of excellence 
applied by the evaluators, these composite evaluations are proving 
to be so detailed and helpful that we have determined to share them 
with the abstractors as well as the evaluators. 




(2) providing feedback to the individual evaluators. At times the 
evaluators have applied standards which are Clearinghouse- 
specific and not enumerated in any ERIC systemwide guidelines. 
Some evaluators are taking the project as an opportunity to 
comment on indexing and cataloging problems. The Documents 
Coordinator has been writing brief informal notes to point 
out when a criticism is not valid in terms of ERIC policies. 
Examples of these notes are also attached. 



As can be expected these two steps involve a areat deal of time; especially 
when the comments of the evaluators must be validated using the original 
documents and abstracts, when the composite evaluations are seen as constructive, 
purposeful feedback, and when the comments to the evaluators are necessarily 
diplomatic and forceful. 
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The amount of time required, though well spent and resulting in potentially 
useful products, is frustrating. It is important to provide feedback to the 
evaluators as quickly as possible, to insure that the project does not lose 
momentum, and to accomplish as many series of evaluations as possible. 

However speed is not possible; each composite evaluation requires at least 
an hour, with this time extended when the evaluations are incomplete or 
contradictory. At this point, 15 abstracts, the work of 5 Clearinghouses 
remain to be processed. 

As soon as these abstracts are dealt with, the Clearinghouses will receive: 

(a) the 12 evaluations of abstracts they submitted in March, with comments 
about evaluation quality, appropriateness, etc.; 

(b) composite evaluations for each of these abstracts to be compared with 
their own evaluations; 

(c) composite evaluations for each of the 3 abstracts from their Clearinghouse 
that were assessed; 

(d) a new assignment of abstracts to be evaluated within one month. The new 
assignment will consist of eight abs'racts, a reduction to allow the Documents 
Coordinator to reduce the amount of time spent in preparing the composite 
evaluations . 

(e) whatever Abstract Evaluation Project forms and instructions that the 
Clearinghouses seem to be missing. 

Additionally, as soon as the second series of evaluations is processed. 

Interim Report ?3 will be prepared to provide an overview of the abstracting 
and evaluation problems revealed so far by the project. 



B. 

VARIOUS VERSIONS OF THE ABSTRACT 
USE. 



EVALUATION CHECKLIST AND INSTRUCTIONS FOR 



ABSTRACT EVALUATION FORM -- INSTRUCTIONS FOR USE 



OVERALL INSTRUCTIONS 



* The original document must be used to complete the evaluations. 

* Comments, in addition to being specific and constructive, must be legible 
and understandable. 

* While comments are required to support poor ratings, comments that point 
out when something is well done are also welcome. 

* The evaluation form is based on the Abstracting section of the ERIC 
Proces sing .Manua l . It may be useful for those unfamiliar with this 
section of the EPM to read it before attempting to use the form. 

ABSTRACT IDENTIFICATION INFORMATION 



Clearinghouse include both Clearinghouse acronym and 

Accession No. accession number. 

Abstractor -- indicate the abstractor's initials or that 

the abstract is an author abstract or an 
edited author abstract. Evaluate author 
abstracts as you would any other abstract. 

Document Type -- provide numeric or descriptive identification 

of the document type (e.g., 143 or Research 
Report ) 

Position -- indicate the job title of the evaluator. 

A . SELECTIO N O F DOCUMENT TYPE 

According to the ER IC Processing Manual , selection of the abstract type "depends 
upon the format or structure of the document and the author's organization of the 
information. " An informative abstract should be used for a document that has "a 
developed thesis that can be summarized accurately. 11 An indicative abstract should 
be used for a document that is "varied, broad, or designed for reference purposes 
and cannot be summarized without distortion." An informative/indicative abstract 
is best used if a "document has many tables, graphs, charts, attachments, appendices, 
etc., along with substantive text." 

Effective use of each of these abstract types requires that the reader never be 
confused as to which type of abstract is being used. In the case of an informative/ 
indicative abstract, the reader should always know which words express the author's 
viewpoint and which express the viewpoint of the abstractor. 

B. STYLE AMD VOCABULARY 

Circle the number that best reflects the intelligibility, logical structure, etc., 
of the abstract. In many cases you will be called upon to make subjective judgements 
about the understandabi 1 i ty of the abstract or the terms it uses. Some concern 
has been expressed that this kind of judgement will be difficult to make as we 
deal with subjects outside of our scope area. Indeed, some latitude must be 
given based on the subject expertise of the abstractor and the knowl edgeabil i ty of 
the intended user. However, if you, as an informed ERIC user, have no idea about ■ 
the subject of an abstract, the abstract could probably have been clearer. 




COMMLMTS: 



In this section, make any general or specific comments you have on abstract type 
or style a nd vocabulary. If you evaluate some aspect of the abstract as less 
than adequate, please comment specifically on why this was so. 

C. INFORMATIONAL CONTENT FOR SPECIFIC TYPES OF DOCUMENTS 




While the Proce ssing Manual permits Clearinghouses to assign Lip to three 
pubtypes to a document, in most cases one pubtype is more indicative of the 
whole document than the other(s). For example, many research reports (143) 
include a literature review (070) and a questionnaire (160). All three pubtypes 
may legitimately be assigned. However, the pubtype 143 is most reflective of 
the document as a whole. When completing this section of the form, try to decide 
upon the most applicable pubtype and evaluate the abstract in terms of on<e set of 
content areas. If it is not feasible to evaluate the abstract according to a 
single pubtype, use as many of these pubtype-speci f i c sets of criteria as are 
appl icable. 

The P rocessing Manual provides guidance for the informational content of abstracts 
for only selected types of documents. In order to avoid the necessity of an "other 
pubtype" category, we have provided some guidance for the document types that the 
Processing Manual does not cover. When dealing with these document types (marked 
by an asterisk on the form), please feel especially free to make use of the "other 
areas" space provided at the end of the list of content areas. Use this space 
whenever necessary to indicate a subject the abstract should have reflected. 

D . A BSTRACT AS A WHOLE 

How long Is the abstract? Is this length appropriate? This word count will 
enable you to gauge how much more information could be expected of the abstract 
given our 200-word limitation. In this way we can avoid making unrealistic 
demands on the abstract. If an abstract has used its 200 words well to cover 
important aspects of a document, it is unfair to make an extensive list of other 
topics that should al so have been included. If the 200 words are not used or 
are used ineffectively, it is fair and important to tell the abstractor what 
other information should have been included. 



kh_e the important topics covered In the. document adequately xeilectexl in the 
abstract? While the abstract cannot completely cover every topic included in 
a document, all i mportant topics should be reflected. If you feel a significant 
aspect of the document has been omitted in the abstract, please indicate so in 
the comments section. Your responses to Section C of the form may help you to 
focus your comments. 



1.6 all o & the intannatlcn In the abstract accurate.? Please verify all information 
provided in the abstract by checking the original document. 



How would you rate the abstract cls a whole? Your overall rating of the abstract 
should be reflective of the e w aluations you have given to its choice of abstract 
type, style and vocabulary, a, id informational content* In cases where your over- 
all rating is not supported by your racings and comments on the rest of the form, 
please explain why. It is altogether possible that an abstract achieve satisfactory 
ratings in specific areas and still have overall failings. It is also possible that 
an abstract may do an excellent job of representing the document and still have 
problems in its style and vocabulary or choice of abstract type. In this case the 
overall rating may justifiably be higher than ratings on Sections A and B. 



ABSTilACT EVALUATION FORM 



(WORKING DRAFT) 



ERIC Document No. _ 

Clearinghouse Accession No. 
Abs t ractor 



Evaluator/Cl ear i nghouse 
Document Type 



Yes/No 



A. Selection of Abstract Type 



ff 1. Is the type of abstract (informative, indicative, mixed) 

appropriate to the document? See EPM p9 -10. 



#2. Is the reader always certain as to whether the viewpoint 
expressed In the abstract is that of the author or that 
of the abstractor? 



Comments : 



B. Style and Vocabulary 

ft 3- Is the abstract a single, coherent paragraph containing 
connected sentences each following the other in logical 
order? 



ffb , Does the abstract have a topic sentence, i.e., some 

coherent statement of the document's major thesis from 
which the rest of the abstract develops? 



#5. Are complete sentences used? 



ft 6. Does the abstract avoid confusingly long and complex 
sentences ? 



111. Does sentence length vary as much as possible?. 



tf 8. Are new or technical terms and abbreviations or acronyms 
de f i ned? 



tf 9. Is the abstract clear, so that it can be read with 
understanding by all users of the ERIC system? 



#10. Is the abstract free of grammatical or spelling errors? 









C, Informational Content (General) 



Yes/No 



If 11. Are all Important topics covered In the document reflected 
in the abstract? 



Is the abstract weighted in the same proportion and with the 
same emphases as the document? 



ff 13 - Is the reader able to find in the document the full and 

corresponding information mentioned in the abstract? 



comments : 



Informational Content (Specific Types of Document^ ) 

#}ha. If written for a research or technical report , does the abstract 
contain objectives and hypotheses of the research, methodology, 
findings and emphasized conclusions, implications or recommenda- 
tions and limitations? 



If written for a s peech or presentation , docs the abstract con- 
tain a statement of the premise or thesis, development or ideas 
or arguments with proofs and major supporting facts, and 
concl us i ons? 



If written for a program description or admin i st rat i ve report , does 
the abstract contain information on the objectives of the program; 
the target population; types of Special teachers and other personnel 
Involved; total number of students or classes Involved; means used 
to carry out the program; special methods used to overcome problems; 
progress to date on the program? 



if written for a textbook or other instructional materials , does the 
abstract deal with the objectives of the text, the general nature 
of the subject matter, and special methods used in meeting objectives, 
such as illustrations, exercises, and accompanying activities? 

If written for a curriculum or teacher guide , does the abstract 
indicate the guide's subject and grade level, specific course 
or program objectives, methods used, and whether supplementary 
activities and materials are suggested? 

#l4f. If written for a bibliography or directory , does the abstract 
deta i 1 subjects considered; the presence of annotations; 
dates of materials included; types of materials listed; 
potential users or target audience; and the number of entries? 

If written for a mu 1 1 i pie -author work , does the abstract state 
the subject and scope of the collection and provide a narration 
of the table of contents, including authors and titles? 



'OPT AVAIfABLE 



Yes/No 

tt lAh. If written for a test or measurement materials , does the '• 
abstract detail the pu r po s e on d o b j e c t i v e of the test; 
forms and levels of the test, including length and 
whether verbal or non-verbal; group to be tested; methods 
of testing, scoring, and grading; normative data; and 
information concerning reliability, validity, and test 
development? 



ABSTRACT EVALUATION FORK 



EPIC Document No. 



Eva 1 ua tor/ Cl oar i nghouse 



Clean nghouse 
Accession No. 



Position 



Abstractor 



Document Type 



A . Selection of Abstract Ty pe 



1 . 


Is the type of abstract (information, indicative, 
appropriate to the document? 


mi xed) 


Yes 


— 


No 


2. 


If mixed, is the transition between styles smooth 
cl ear? 


and 


Yes 


— 


No . 


3. 


Can the reader determine whose viewpoint is being 




Yes 




No 




expressed (author or abstractor)? 











B . Style and Vocabulary 

The abstract: 

1 . is coherent . 

2 . is structured logically. 

3. is grammatical ly and orthographical ly correct. 

4. is ur.derstandabl e to the nonspecialist. 

5. has a topic sentence. 




12 3 4 
12 3 4 
12 3 4 
12 3 4 
12 3 4 



6. uses complete sentences. 1 2 3 

7. defines new or technical terms, abbreviations, or acronyms. 1 2 3 

8. avoids monotonous sentence patterns. 1 2 3 

COMMENTS: (please comment on any item rated less than satisfactory) 



4 

4 

4 
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Compl etely 







Informational Content for [ 
Specific Types of Documents! 

Please rate abstract: 

1 - does not cover area 

2 - covers a rea si Ightly 

3 - covers area adequately | 

4 - covers area completely ' 



Objectf ves 
Hypotheses 

Research Methodology 
Research Findings 



Concl usi ons 



Impl i cations 



Recomnendations 



Premise or thesis 



Development of ideas 



Major supporting facts 
Target population 
Personnel involved 
Number of students, etc 
Methods used 
Progress to date 
General subject matter 
Supplementary materials 
Da tes of na terial s 



Presence of annotations 



Number of entries 



Scope of collection 
Authors included 



Titles Included 



Forms and levels 



Rel labil ity/val idlty 
Test development 
Appendi ces 
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COMMENTS : 
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D. ABSTRACT AS A WHOLE 



1 . Are the important topics covered 
reflected in the abstract? 

Is the abstract weighted in the s 
and with the same emphases as the 

3. Is the reader able to find in the 
full and complete information men 
abstract? 



4. How would you rate the abstract as 
Excellent Good Fair 



COMMENTS: 





ABSTRACT EVALUATION FORM 



RIC Document No* E_D 

.learinghousc 
accession No 



\bs tractor 
Document Type 



IAj.. Sele ction o f Abstract Type 

I). What is the abstract type? (Circle one) 

J2. Is the type of abstract appropriate to 
the document? 

|3. If mixed, Is the transition between 
styles smooth and clear? 

4 . Can the reader determine whose viewpoint 
is being expressed? 



Eva 1 uator 



Position 



Clearinghouse 



I n f o rnia live 


Indicative 


Yes 


No 


Yes 


No 


Yes 


No 



B * St yl e jm d Vo ca bul ary 



1* The abstract is intelligible. 

2. The abstract has a logical structure. 

3. The abstract uses correct spelling and grammar. 

4. The abstract is understandable to the nonspecialist. 

5. The abstract has an adequate topic sentence. 

6. The abstract uses complete sentences, 

7. The abstract explains new or technical terms, abbreviations, or 
acronyms. (Answer only if applicable to document in hand.) 

8. The abstracts uses varied sentence patterns. 

COMMENTS; (Please comment on any item rated less than "adequately,") 
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Not at all 




I 



C. Informational Content for Specific Types of Documents 

Please rote the abstract in the section corresponding to the most applicable Pubtype for the document. 
In most cases you will only mark one section. Use the following scale: 

1 - does not cover area at all 

2 - covrrs a rea si Inhtl y 

3 - covers area adequately 

4 - covers area completely 

n/a - not applicable to document in hind 



jOLUCTED WORKS (020, 021, 022) 

Bubjcct/scopp of collection 
jresentat ion of authors 
listing of titles 
Sol Uc live presentation of 
SUrticles in collection 
Ithcr areas 



isstPJAnoris/TH r srs (oto.oti ,o 

iteratune review 
bjectives of study 
ypothesns of study 
esearch methods 
esearch findings 
oncl usions 
mol ications 
imitations 
sccre-enda lions 
ther areas 



||j] OES (050, 052, 055) 

Subject 

grade level 

H'ojec lives of guide 

jSr.hods used 

Bjggostcd activities and 

Material s 

pther areas: 



INSTRUCTIONAL MATERIALS (051) 

Hi . . 

ajee ti ves 
irget population 

i jtject matter 
it hods used 

jgg-sted activities and 
material s 
ther areas 



SI Of? I CAL MATERIALS (060) 
Subject 

P‘thods/sources used 



her areas 



[ i 
[ i 
[ ] 

[ ] 

.2.1X3) 



*IKrOf!!!ATI0:i AMUSES (070, 071 ) 

subject /scope 
materials covered 
cone 1 usions 
other areas 



‘STATISTICAL DATA (110) 

subject/ scope 
objectives 
audi cnce 
fl ndings 
other areas 



necessary background 
premise or thesis 
development. oT ideas 
major supporting fact 
conclusions 
other areas 



REFERENCE MATER I Al S (131, 13?) 

subject matter 
reference to annotations 
dates of material s 
types of material s 
potential users 
number of entries 
other areas 



DESCRIPTIVE: REPORTS (141 ) 

program objectives 
target population 
personnel involved 
number of participants 
methods used 
progress to date 
evaluation of effectiveness 
other areas 



:VALUATJve REPORTS (142) 



[ 


i 


subject, of evaluation 


[ i 


[ 


1 


evaluation objectives 


[ ] 


[ 


i 


target fopulation 


[ 3 


[ 


] 


personnel involved 


[ ] 


[ 


i 


number of participants 


[ ] 


[ 


j 


methods used 


[ 1 






progress to date 


[ 1 






evaluation methods 


[ ] 


[ 


i 


evaluation criteria 


[ ] 


[ 


] 


evaluation results 


[ ] 


[ 


i 


other areas 


[ ] 


[ 


i 




r ] 


[ 


i 




[ i 


[ 

[ 


i 

] 


RESEARCH/7FCHHKAL REPORTS 


(143) 


r> 1 




research objectives 


E ] 






research hypotheses 


E ] 


[ 


i 


research methodology 


[ ] 


[ 


i 


findings 


[ ] 


[ 


i 


conctusi ons 


[ ] 


c 


] 


inpl ications 


f ] 


[ 


i 


1 imi tat ions 


f i 

L J 


c 


i 


appendices 


[ 1 


[ 


] 


other areas 


[ ] 


[ 


i 




E ] 








[ ] 


[ 


i 


TESTS/EVAE'JAl 10'i INSTRUMENTS (IS 


[ 


i 


purpose 


[ J 


[ 


i 


forms and levels 


[ 1 


[ 


t 


group tc be tested 


[ ] 


[ 


3 


testing, scoring, 




r 


] 


grading methods 


[ 1 


• [ 


] . 


normative data 


[ ] 


[ 


] 


test development 


[ 1 


[ 


] 


other areas 


[ ] 








[ ] 








[ ] 


[ 


l 






[ 

[ 


] 

i 






[ 


5 






[ 


] 


Use the space provided to 




[ 


] 


indicate content areas 




r 


] 


Other than those suggested 




[ 


] 


that were covered or should 


[ 


] 


have been covered in the 




[ 


] 


. abstract. 
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Form pubtypes (010, 030, 150, 
and 170) are not included, as 
they should t»“ used in conjunc- 
tion with other pubtypes. 

Use other assigned pubtypes. 

•These document types are not 
covered by the ERIC Pro c essi ng 
Man ual . Tne content areas 
provided here are suggestions 
of things to look for. 



IT. Abstract as 3 Wno le . 

1.. Hovj , 1 ony is the abstract? words 

Is this length appropriate? Yes No 

COMMENTS: 



2. Are the important topics covered in the document adequately reflected in the 

abstract? Yes No 

COMMENTS: 



3. Is all of the information in the abstract accurate? Yes No 

COMMENTS : 



4. Is the reader able to find in the document the full and complete information 

mentioned in the abstract? Yes Mo 

COMMENTS: 



5. How would you rate the abstract as a whole? 

Excellent Good Fair Marginal Poor 

COMMENTS: 

13 : ' r ' r c op v AVA ! LABI. E 



f Continue comments on other side) 




c. 

SAMPLES OF FEEDBACK FROM THE CLEARINGHOUSES TO THE PHOT TESTING OF THE 
CHECKLIST. 
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November 6, 1981 



Dr. Arlhur M. Cohen 
Director 

Ms. Anita Colby 

Documents Coordinator 

ERIC Cli. for Junior Colleges 

University of California 

96 Povcll Library Building 

405 Hilgard Avenue 

Los Angeles, California 90024 

Dear Art and Anita: 

Thanks for the opportunity of seeing the new forms on the abstract 
quality project. Some of our staff have done the trial abstracts 
and Jim, Regina, Syd , and I have discussed the procedures at length. 
Here are comments and suggestions. 

I. We think that going from a two-point to a f our-point scale 
on most of the items is an improvement. We are split on whether three 
or four is better. But if you do keep a ft ur-point scaLe, we all feel 
that it would be better to omit the middle terms, since "slightly" and 
"adequately" are subject to very different interpretations. Maybe 
"Not at all" and "Very" for the end-points, without labelling the mid- 
point (s) would make an easier task for evaluators. On item D-4 , we 
would prefer a four-point scale, omitting "marginal." The two-point 
scale for A is ok. 



2. After the B section, comments are requested — good — but the 
referent for "less than satisfactory" is rot made clear. On a four- 
point scale, how about asking for comments on items ranked 1 or 2? 



3. B-5 is a bit difficult to answer. A yes-no answer for whether 
there is a topic sentence would be appropriate; then, if there is one, 

a scaling of its quality. 

4. You haven’t made provision for reporting of word length. I 
know you don’t think this is important, but if one clearinghouse has 
abstracts averaging 50 wolds and another 150 words* I don't believe 
this can be attributed to differences in subject matter. Either one 
clearinghouse is too wordy or the other not giving as much information 

_ « ^ ns would be useful, and I suspect, the latter. riu.m.-r 

srousoRirti, ^>lUTl funding 
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Page 2 

5. Il is usi.ful to have. ;m overall judgment of the evaluator, as 
in D-4; but some oLiier glubaL measure or measures of quality would bo 
useful. The evaluations will not be useful unless there is sone method 
of analysis and summary. What will this bo? 

6. You haven't included anything about the purposes of the study, 
nor how clearinghouses will be asked to participate in the project 
(voluntary or required, specified or negotiable number of evaluations, 
and single or replicated evaluations) — which we suggested previously. 

It would be usefuL to inform people about these aspects of the project. 

The evaluations which you requested are enclosed. Each one took about 
15 minutes. We would be willing to do 12 to 15 evaluations per month. 

Best wishes to you and to the project. 

Cordi al ly , 



r is sett 
Executive Director 



0 N 

-Irving Mor' 
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Clearinghouse on 

Elementary and Early Childhood Education 



Collago of Education 
University of Illinois 
1310 South Sixth Street 
Champaign. Illinois 61820 
(217) 333*1 386 



November 2, 1981 



Dr. Anita Colby 

ERJC Clearinghouse for Junior Collges 
University of California 
96 Powell Library Building 
405 IliJgard Avenue 



Dear Anita, 



Following are several suggestions and comments concerning the Abstract 
Evaluation Form. These comments and suggestions arc fairly persnickety 
and I believe that they deserve your serious consideration. First, 
though, may I congratulate you on the form, contents and organization 
of the evaluation materials. It is clear that you have worked hard on 
a demanding task. 

Abstract Evaluation Form 
A. Selection of Abstract Type 
A2 should read 

If mixed, is the transition between informative and indicative 
st vies smooth and clear? 



B. Style and Vocabulary 



It is not clear how items 131 and B2 are dijferent. My Oxford American 
Dictionary defines COHERENT as ad j . 1. cohering. 2. connected logical- 

ly, not rambling in speech or in reasoning. COHERE is defined as 
to stick together, to remain united in a mass, the particles cohere . 
Since definition 1 is not applicable, that leaves us with definition 2 
of coherent. And definition 2 ot coherent is nearly the same as item 
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If B5 is adequate then it is completely correct, (in ref. to grammar 
and spoiling). 

1 B'l can he operationalised a way from a surmise about the understand- 

ing of the nonspeeia 1 i st by listing the most important of those 
■ features likely to confuse the nonspecialist. Such as, 

| 4. is free of convoluted sentences and esoteric terms. 

| B5 is an item calling for an absolute judgment, yes or no. One 

j cannot judge whether or not a topic sentence is slightly included. 

i 

| B6 could be revised Replace "uses" with "consists of M ... .Again, 

I the presence of one incomplete sentence makes the abstract inadc- 

* quatc and the absence of an incomplete sentence makes the abstract 

I totally adequate. So, this looks appropriate for a yes or no 

I judgment. 

I OTHER: Change "satisfactory" to "adequate" in the Comments section 

on page one of the Abstract Evaluation Bonn. The line should read. 
Comments: (please comment on any item rated less than adequate) 

This way we don't have to slice a satisfactory with an adequate. 



C. Informational Content for Specific Types of Documents 

Some of these categories are vague. To decipher them one has to 
search through the list and identify the document type with which 
they are associated. You might suggest to evaluators that they 
do this if they are uncertain about what an item, such as 'general' 
subject matter', refers to. When the phrase 'general subject matter' 
is juxtaposed with Textbook/Inst. Material, Curriculum/Teacher 
Guide, arid Bibliography/Di rectory (ALL documents dealing primarily 
with subject matter in a restricted sense) then the meaning of the 
phrase becomes clear. Likewise with several other phrases on the 
list . 



D, Abstract as a Whole 

Item D3 reverses the orientation of abstractors and editors. They 
are usually looking to see jf the contents of the document can be 
found in the abstract. Here, the evaluator is requested to sec if 
the contents of the abstract can be found in the document. Since this 




is so similar in wording and so different in procedure, maybe a 
NOTE: statement is advisable. Such as, 

Note: See i f you can find all of the contents of the abstract 

i 1 1 the document, 



Now it's Time to Say Goodbye 

1 aru happy that 1 have had a chance to got some of my comments 
back to you in time lor them to be useful. The earlier agreement to 
blfS them broke down due to the fact that liliiS was turned off, I got 
a flu virus, and the material you handed out at the NTech Mtng and my 
comments became lost. Excuses". Hope you like this batch. My impartial 
assessment of this evaluation project is that it is interesting to do, 

(if time consuming) anJ might turn up sumo interesting findings. 

Sincerely yours, 




Ron Hutchison 
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Dear //3//: 

Here is the final version of the checklist and instructions for partici- 
pation in the abstract evaluation project. This project was initiated 
by Central ERIC in response to concern that the abstracting in the ERIC 
database is uneven. Central ERIC asked us to design and implement a 
study that would evaluate currently written abstracts in the system 
and serve as a training experience for abstractors and editors at the 
Clearinghouses. Your participation in the pilot testing and development 
of the study design has assured that the project will be an interesting 
and valuable one. 

The project is based on the following premises: 

(1) ERIC abstracts can only be evaluated on the basis of the guidelines 
in the ERIC Processing Manual ; 

(2) ERIC abstracts can be assessed most effectively by ERIC personnel 
as these are the people most knowledgeable about ERIC abstracting 
and their feedback will have the most credibility with the Clearing- 
houses being evaluated; 

(3) an ongoing feedback mechanism will be part of the project design; 
and 

(4) the project will continue long enough for the impact of the evalu- 
ations to be demonstrated in the abstracts appearing in Resou rces 
in Education . 

On the basis of the pilot test, it seems reasonable to expect each 
Clearinghouse to evaluate sixteen abstracts per month, given that each 
assessment should take approximately 30 minutes to complete. This will 
require the Clearinghouses to devote eight person hours per month to 
the project. As this task can and should be divided among several 
Clearinghouse staff members, representing both processing and user 
services personnel, this should not prove an undue burden on the 
Clearinghouses. 
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Each month you will be assigned sixteen randomly selected abstracts and 
asked to return the completed evaluations within three weeks. This will 
allow two weeks to complete the task and a week for the completed forms 
to be returned in the mail. Photocopirs of the assessment will be sent 
to the Clearinghouses who prepared the original abstracts. Processing 
coordinators, abstractors, and editors should carefully examine the 
evaluations and initiate corrective measures whenever necessary. Because 
of this very important training aspect of the project, we must continue 
to stress the importance of careful, thorough, specific, and constructive 
eval nations. 

Using this procedure we should be able to evaluate the abstracts from 
seven consecutive months of Reso urces in Education before the project 
is completed in July 1982. V/e have provided for the possibility of 
mid-course design corrections and will reevaluate the project after 
the third month to determine whether any changes should be made in the 
project design. 

Our final report, which will be submitted to Central ERIC in August. 

1982, will discuss abstracting in the system as a whole and will not, 
unless there are serious and uncorrected problems, identify specific 
Clearinghouses. Central ERIC intends to use the results of the project 
for systemwide training at future national and regional technical meetings. 

Thank you for your cooperation and support. 

Si ncerely , 



Arthur M. Cohen 
Di rector 

X4ERIC/B 
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Dear //^//: 

Enclosed please find the second set of ED numbers for the abstracts to 
be evaluated by your Clearinghouse. Preliminary analyses of the first 
set of evaluations have revealed some wide divergences among the Clearing- 
houses on the qualities of a good abstract. The evaluations v/e received 
showed the application of very different standards of excellence and 
different amounts of effort put into the assessments. Those very incon- 
sistent evaluations will not be especially useful to the people who 
wrote the abstracts and will not be returned to thorn. 



With the intention of achieving a greater level of consistency in the 
evaluations, certain changes in the study methodology have been made. 

In the next few months, each abstract will be evaluated by four different 
clearinghouses. If these evaluations continue to show the same variations 
found in the first month’s assignment, the Junior Colleges Clearinghouse 
will prepare a composite evaluation for each of the abstracts, based on 
the input received. This composite assessment will be sent to the 
evaluators to help them recognize when the standards they have applied 
are in-house criteria, when they have missed an important problem in an 
abstract, or when their expectations are too demanding or too lenient. 

These changes in the evaluation methodology should have no impact on the 
amount of time and effort required of your Clearinghouse personnel. 
However, as we will not need an especially large sample to coordinate 
evaluation efforts, we are able to reduce the number of abstracts in 
your monthly assignment to twelve. 

Thank you for your continuing support of the project. 

Sincerely, 



Anita Colby 
Documents Coordinator 

X4CRIC/E 
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Evaluators; David Tipton (CF) 
S yd Meredith (SCJ) 

A. Colby (JC) 

Abstractor: PC ~ SE 



A. Selection of Abstract Type 

The abstract takes a primarily indicative approach, although the 2nd and 
3rd sentences are informative. The mixed node was considered by the evaluators 
to be appropriate to the document; however, two evaluators felt the mixed mode 
could have been used more effectively. 

B. Style and Vocabulary 

The evaluators identified a number of stylistic problems with the abstract 
which they felt made it rather difficult to follow. 

All three evaluators felt that the abstract was slightly intelligible, due 
primarily to the long and confusing sentence patterns used and to the somewhat 
confusing way the sentences stated the scope of Project Synthesis. 

Two of the evaluators felt the abstract could have been more logically structured 
had it followed the format of the document more closely. 

The abstract was considered to use correct grammar and spelling by two of the 
evaluators. One evaluator felt that the parenthetical statement in sentence "1 
would have been clearer had it included a verb. 

All three evaluators found the abstract to be only slightly understandable to . 
the nonspecialist, pointing out a lack of explanation of "the national survey," 

" the focus groups," and "four student goal clusters." 

Two of the evaluators felt that the abstract used completed sentences, with 
one evaluator reiterating the comment that the parenthetical statement could 
have used a verb to increase its clarity. 

Two evaluators noted that the abstract consisted of three sentences, two of 
which were identical in structure and awkwardly long. 

C. Informational Content for Specific Types of Documents 

None of the evaluators found the content areas specified on the form especially 
applicable to the document. 

D. Abstract as a Whole 
1 . Length (1 51 words ) 

Two of the evaluators felt that the abstract was too short and that the 
additional 50 words alloted by the processing manu: could have been used to 

provide additional information. 
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2. Are the important topics covered in the document adequately reflected in the 
abstract ? 

Two of the evaluators felt that several important topics were not covered; 
one evaluator felt that while the important topics were reflected in the abstract, 
it failed to provide the user with a picture of the document itself. 

All evaluators felt that the abstract was accurate and that th* information 
it contained was to be found in the document. 

Two of the evaluators rated the abstract as poor; one rated it as fair feeling 
that despite its problems the abstract provided a reasonable representation of 
the topics covered in a form that was not entirely unreadable. 

One evaluator offered the following alternative as a method for dealing with 
the document (this alternative has been modified so; ewhat to address the comments 
and criticisms of the other evaluators): 

ABST : : Project Synthesis was a research study designed to synthesize and interpret information 

found in a database composed of: (1) the results of an extensive review of research on 

science education ( 2 ) a research study assessing the status of precollege science education 

(3) case studies in science education; (4) a national survey of science education; (5) a 

review of journal literature; and (6) an analysis of science texts. Using this data, five 

working groups sought to assess precollege science education arid recommend future activitiep 

The five groups focused on the tasks from the different perspectives of biological sciences!; 

physical sciences; inquiry in school science; science, technology, and society in secondary 

schools' and elementary school science. The first two sections of the monograph on Project 

Synthesis describe its background, purpose, organization, and procedures. Sections III through 

VII contain edited versions of the five working group reports. Each report evaluates scien:e 

education in the focus area with respect to clusters of student goals related to personal 

needs, societal issues, academic preparation, and career education awareness. These reports 

j use a discrepancy model to describe the actual and ideal states of education in their focus 

| area and discuss the descrepancies between these states. After Sections VIII and IX offer 
I ~ 

! recommendations for the science instructor and a bi bl iograpliy , Section X lists priorities 
for improving science education. (SM/DT/AYC/KL/HB ) 3F,..- * * YAW-ABLE 
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Evaluators : 



Paula Wagemaker (PS) 

Mary Niebuhr (FL) 

Stan Helgeson (SE) 

Francie Gilman (SP) 

Maryellen LoEosco (UD) 

Anita Colby (JC) 

Attendees at the Eastern Regional Technical Meeting 
ABSTRACTOR: NEC (RC) 

A. Selection of Abstract Type 

The evaluators all agreed that the abstract was written in the 
informative mode. Most agreed that the mode was appropriate; 
one evaluator, however, felt the abstract should have been 
written in the indicative mode. Most were able to determine 
whose viewpoint was being expressed; one indicated that the 
reader could not determine this. 

B. Style and Vocabulary 

For the most part the abstract was rated as at least adequate 
in the areas of style and vocabulary. There was general agree- 
ment about certain problems: 

Most evaluators felt that the first sentence was too 
long. Some found it confusing. Even those who did 
not find it confusing felt that it should have been 
broken into at least two sentences, e.g., "During a 
1979-1980 pilot project, 13 nine-week residencies by 
professional artists were sponsored in 10 eastern 
Oregon school districts. Eastern Oregon College served 
as liaison during the project, the Northwest Area 
Foundation . . . " 

Several evaluators felt that the abstract lacked a 
logical structure. They indicated: ,! the abstract 

tends to leap from topic to topic"; "project procedures 
should have been treated before the outlook for the 
future, which should, logically, he Ms t M ; "the transi- 
tion between the topic sentence and second second was 
ackward. School funding reference was not clear to 
me . " 

With respect to the topic sentence, the most frequent 
comment concerned its length. Some evaluators felt 
that it also distorted the tone and content of the 
document . 

One- evaluator commented that the reference to the "State 
Arts Commission guidelines" was very vague and that as 
the document failed to give any information on these 
guidelines, the abstract might have done better not to 
include this reference. 
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C. Informational Content for Descriptive Reports 



A fairly wide rang., was represented in the evaluator's ratings of 
content coverage: 



Adequately 

covered 



(3/4) 



Not 

Adequately 

covered 

( 1 / 2 ) 



Not 

Applicable 



program objectives 
target population 
personnel involved 
number of participants 
methods used 
progress to date 
evaluation of effectiveness 



2 

3 

12 

3 

2 

2 

0 



7 

8 
0 
1 
5 

3 

4 



0 

0 

0 

9 

-t- 

1 

2 



(The reason these rows each have different totals is the inconsistent 
collection of responses during tech meeting discussions) 

D. Abstract as a Whole 



1. Length (169 words) 

Two of the evaluators were satisfied with the length; 3 felt 
it should have been longer; 2 felt it should have been shorter. 
Those that recommended shortening the abstract commented: 

"This is a difficult document to abstract concisely. Some 
information in the abstract was brief to the point of 
inaccuracy or tantalization, and could have been left out." 

"The whole abstract could be shortened- -indicat ive mode which 
briefly states what the document is about (or as much as the 
abstractor can ascertain what the document is about). would 
have been better I think. K 7 ould probably have reduced the 
unwieldly and confusing nature of the abstract." 

2. Coverage of Important Topics 

Three evaluators felt that the abstract covered the document's 
important topics. Three indicated that the important topics 
had not been adequately reflected. They noted: 

"Some of the topics are overrepresented and others not 
represented at all. The abstract goes off ona fiscal 
tangent and I arn left wondering what the artist residency 
program is about and how it relates to ruriil schools." 

"The actual case histories could/should have been included to 
f firm it up* 

"Among the topics not covered are the unique features of the 
program, the Regional Arts Council, variations among districts, 
and 'examples of the use of residency programs. Perhaps the 
abstractor might have teem able to mention those aspects of 
the document and at the same time give an indication of how 
briefly they are noted in an indicative mode." 
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3. Accuracy 

The evaluators agreed that the abstract was accurate. 

4. Is the reader able to find in the document the full ana 
complete information mentioned in the abstract? 

While most evaluators agreed that the abstract contained 
no information that was not in the document, some felt that 
the over-emphasis in the abstract on the financial aspects 
of the document provided a distorted image of the document’s 
actual content. 

5. Overall ratings 

3 good; 2 fair; 2 marginal; 1 poor 
Comments : 

"Tfye abstract tends to over -emphasize the funding and deemphasize 
the program itself, although the document includes little 
concrete information about the program." 

M I do not think the abstractor is entirely to blame for the 
status of the abstract. I found that the purpose of the 
program was evident only implicitly in the document and I 
know there is always a danger of trying to fit such implicit 
information into a form that could indicate to the reader 
of the abstract that what is written M 'tre is in fact what 
can be found in the document. Either u.he author of the 
document should have been contacted for clarification or 
the document should have been rejected. As a reader, I want 
to know what the program is about and what it means or says 
for educational practice in rural schools. I didn't get that 
information. Because I don't know what the program's purpose 
and intent was from the abstract I was left wondering how the 
information given in the abstract fitted in with the program 
as a whole. Essentially, the abstract leans too heavily in 
one direction giving particulars about the program without 
addressing what the program was about in general." 

’’The abstract's overall weight and order to not adequately 
reflect the document, though the document itself was no 
peach. n 

"The abstract did not get across the excitedment about the 
project, the 'plug' the author is putting in for the school 
districts to fund this program. I thought this should have 
been included . " 
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EVALUATIONS 



OF ED 205 273 



EVALUATORS: Helen Mamarchev (CG) 

Anita Colby (JC) 

David Tipton (CE) 

John Brosseau ( FL ) 

ABSTRACTOR: MR -- PS 

A. Selection of Abstract Type 

The indicative mode selected for the abstract was considered 
appropriate for the document and to have been used effectively. 

B. Style end Vocabulary 

The abstract was rated highly by all evaluators in all areas of 
style and vocabulary. One evaluator did comment that repeating 
"the document" in line 12 was unnecessary and that a synonym 
should have been used. 

C. Informational Content for Teacher Guides 

The evaluators agreed that the guide's subject, grade level, objectives, 
methods, and suggested materials had been adequately or completely 
covered. One indicated that the appendices and bibliographies at the 
end of each chapter might have been mentioned, 

D. Abstract as a Whole 

1 . Length (1 30 words ) 

Two evaluators felt that the abstract was somewhat snort and that 
the additional 70 words could have been used to provide additional 
information. 

2 . Coverage of Important Topics 

One evaluator felt that all important topics had been covered. 

The other evaluators collectively noted (though did not necessarily 
agree) that the following areas should have been covered; 
appendices; bibliographies; materials for the Kindergarten - Grade One 
transition; physical space needs of children in the classroom 
environment; contemporary background (first chapter); and glossary of 
terms . 

3. Accuracy 

The evaluators all found the abstract to be generally accurate, although 
one evaluator noted t.iat the parents were not only helped to understand 
the child's ...development; but are helped to help the child develop. 

r n 

4. All evaluators were able to find in the document the full and complete 
information mentioned in the abstract.. 



5. 



Ratings; 1 excellent; 3 good 
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ABSTRACTING WORKSHOP: 
Draft Agenda and Timeline 




DAY 1 

9:00 - 9:30 
Anita Colby 

9:30 - 10:30 
Harold Borko 

10:30 - 10:45 

10:45 - 12:00 
"‘Betty Rose de Rios 

12:00 - 1:00 
1:00 - 4:30 



"Anita Colby 

Betty Rose de Rios 

^Sydney Meredith 

4:30 - 5:30 
Anita Colby 



Optimizing ERIC Abstracting Quality 



SUMMARY OF ABSTRACTING PROJECT 
THE PURPOSE OF THE ABSTRACT 

The overall purpose of abstracts and their possible functions 
as document surrogates, free text searching sources, or con- 
tent summaries will be addressed. 



BREAK 



INFORMATIVE VS. INDICATIVE ABSTRACTS 

The EPM guidelines will be reviewed in terms of the criteria 
for selecting informative, indicative, or mixed abstract 
types . 

LUNCH 

APPLYING THE GUIDELINES FOR ABSTRACTS : THREE CONCURRENT SESSIONS 

Clearinghouse Preparation : Anita Colby will select 3 documents 

already in the system, one which lends itself best to an 
informative abstract, one indicative, one mixed. A.11 partici- 
pants will abstract the same 3 documents and bring their ab- 
stracts and a copy of the document. In sessions, they will 
compare which kind of abstract they produced for each and 
will discuss T *hy one fits bets. All will take part in ail 
3 sessions. 

Indicative 

Informacive 

Mixed 

REVIEW AND CLOSURE 

The findings of the day will be reviewed and discussed 
and closure will be attempted for remaining problems. 



DAY 2 



9:00 - 12:00 
Anita Colby 



12:00 - 1:00 

1:00 - 2:00 
*Hclen Mamarchev 



2:00 - 3:20 
Anita Colby 



3:30 - 3:45 

3:45 - 5:00 
*Lynn Barnett 



5:00 - 5:30 
Anita Colby 



INFORMATION INCLUDED IN ABSTRACTS 

Part 1: Participants will review the EPM guidelines to 

identify the information which is recommended for inclusion 
in abstracts for some document types. 

HANDOUTS: Perhaps one sheet showing document type and the 

list of information the EPM suggests should be included, e.g., 
(see below) and an example of a good abstract for the same 
document. . 



VIEWPOINTS* POSITION PAPERS (120) 

necessary beckcrcund [ 
precise or thesis L 
development of Ideas [ 
major suoportlng facts L 
conclusions [ 
other aress [ 



] 

] 



! 



Part I : Participants will talk about document types not 

addressed in the EPM and will attempt to put together a list 
of such items. 

HANDOUTS: Same as above. 

LUNCH 

THE POTENTIALS OF FORMATING 

Several clearinghouses have developed formats to apply to 
several document types. These formats establish a set 
pattern to the presentation of info nation, and the abstractor 
modifies the pattern to provide information about a Darticular 
document. A panel of representatives from those clearinghouses 
using this approach with success will discuss the formats 
developed and the cost-effectiveness and efficiency of use. 

PROBLEM DOCUMENTS 

Clearinghouse Preparation : Each clearinghouse will bring a 

sample of a special ’’problem document" and the abstract used 
and be prepared to comment on why it was handled in the manner 
it was. 

SESSION STRUCTURE: After introduction to build the context, 

we will break out into 6 groups for discussion. The following 
people will be asked to act as facilitators for discussion. 
(Since there is no right or wrong answer to these, it was felt 
there would be no need for these people to do a lot of prepara- 
tion): David Tipton, **Dorothv Beling, Ron Hutchison, 

Barb Minor, Barb Wildemuch, Roger Walke. 

BREAK 



r TYLISTIC CONSIDERATIONS AND THE ROLE OF ABSTRACT EDITORS 
Discussion will center about the impact of logical structuring, 
topic sentences, explanations of new or technical terms, and 
varied sentence patterns to the overall usefulness of an 
abstract. Clearinghouses will discuss how they approach the 
editing function: procedures they use, why the steo is important. 



Wrap-up 
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DAY 3 



9:00 - 11:00 

“'Jane McClellan 
Sydney Meredith 



11:00 - 11:15 



SELECTING /TRAIN IMG NEW ABSTRACTORS AND MAINTAINING QUALITY 
CONTROL 

Discussion will center on clearinghouse experiences in 
training and selecting abstractors. Problems and benefits 
associated with in-house and freelance abstractors will be 
considered and methods of balancing quality and productivity 
discussed . 

BREAK 



11:15 - 12:30 
Anita Colby 



ONGOING SYSTEM OF MAINTAINING QUALITY AND STANDARDS IN 
ABSTRACT WRITING 

Clearinghouse Preparation : Participants will come with 

suggestions for ways to do this and will be ready to debate 
the pros and cons of same. A preliminary consensus and a 
recommendation to Central ERIC will be the goal of this 
session. 



1:30 - 3:30 



Steering Committee meets to corsider evaluations of workshop 
and recomiaenda tions of participants , 



* Preparation of Major Session leaders 

One meeting in January 1983 (see time line). 

** Preparation of Concurrent Session Facilitators 

This can be handled through a letter to individuals with instructions on 
the purpose of the session and how to proceed. 
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t racting Workshop Timeline 



September 15, 1982 



October 1, 1982 
October <:0, 1982 



November 1, 1982 
November 15-17, 1982 



January 10, 1983 



February 14-15, 1983 



Directors' meeting announcement and 
discussion of workshop 

Confirm session leaders (A. Colby) 

Clearinghouse Mailing from Steering Committee 
or Central ERIC. Purpose: to build enthusiasm 

and support early by pointing to the objectives 
of the workshop as professional development and 
improvement of the ERIC system. Mailing will 
stress that six Clearinghouses and the Facility 
have participated in the development of the workshop 
and its agenda. The preliminary agenda will be 
sent with the mailing and suggestions/comments 
will be due back to A. Colby November 1. 

Agenda will be revised based on Clearinghouse input. 
Steering committee will be advised and session 
leaders announced. 

Steering Committee Meets in DC. Purpose: National 

Technical Meeting Agenda and discussion of workshop 
agenda . 

Clearinghouse Mailing, Purpose: Instructions for 

exercises and activities to be accomplished prior 
to the workshop. 

National Technical Meeting 



February 16-18, 1983 Abstracting Workshop 
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